Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X....X.......] duration:
332.570393ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:15:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 332.579871ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [.........X.XXXXXXXXXXXXXXXXXX.XX............................] duration:
2.753447216s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:14:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.753462213s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................................X.........] duration:
485.88895ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:14:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 485.905902ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X....X......] duration:
334.758084ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:14:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 334.776598ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
4.688408202s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:14:27 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.68843327s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [..........X.XXXXXXXXXXXXXXXXXX.XX...........................] duration:
2.254311142s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:14:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.254333954s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
5.33041711s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:13:55 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 5.330441974s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................................X........] duration:
464.519372ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:13:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 464.537987ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X....X.....] duration:
317.20474ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:13:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 317.242462ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
4.623275701s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:13:07 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.623295809s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [...........X.XXXXXXXXXXXXXXXXXX.XX..........................] duration:
2.686521768s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:13:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.686535933s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X....X....] duration:
342.811454ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:13:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 342.83037ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
4.978521624s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:12:48 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.978543645s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................................X.......] duration:
481.904846ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:12:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 481.930595ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X....X...] duration:
386.58479ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:12:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 386.724351ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [............X.XXXXXXXXXXXXXXXXXX.XX.........................] duration:
2.651658301s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:12:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.651677426s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
4.557742572s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:12:06 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.557779981s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
4.473741663s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:11:42 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.473758965s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................................................X......] duration:
534.966008ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:11:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 534.991255ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [.............X.XXXXXXXXXXXXXXXXXX.XX........................] duration:
2.243640028s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:11:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.243663972s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X....X..] duration:
445.297875ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:11:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 445.313073ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
4.442066886s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:11:19 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.442093434s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration:
836.057845ms trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:10:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 836.068736ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X....X] duration:
30.000997888s trace:
https://jaeger.dev.storj.io/trace/36cf2dc7bd134cfe
FAIL (2026-01-28 13:10:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001015089s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................................................X.....] duration:
504.222486ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:10:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 504.244578ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
4.965465665s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:10:31 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.965492215s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [..............X.XXXXXXXXXXXXXXXXXX.XX.......................] duration:
2.605127218s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:10:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.605144862s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
884.747346ms trace:
https://jaeger.dev.storj.io/trace/16ac22bfe521b3ea
FAIL (2026-01-28 13:09:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 884.765821ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.901940594s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:09:22 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 6.901961583s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................................................X....] duration:
611.126509ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:09:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 611.14905ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [...............X.XXXXXXXXXXXXXXXXXX.XX......................] duration:
2.664420473s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:09:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.664442374s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
377.567484ms trace:
https://jaeger.dev.storj.io/trace/16ac22bfe521b3ea
FAIL (2026-01-28 13:08:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 377.576821ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...] duration:
586.437621ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:08:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 586.467567ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
4.668866629s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:08:17 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.668889872s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
486.950641ms trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:07:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 486.973414ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [................X.XXXXXXXXXXXXXXXXXX.XX.....................] duration:
2.244249394s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:07:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.244275312s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
359.851021ms trace:
https://jaeger.dev.storj.io/trace/16ac22bfe521b3ea
FAIL (2026-01-28 13:07:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 359.869336ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
6.73931126s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:07:29 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 6.739334752s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [.................X.XXXXXXXXXXXXXXXXXX.XX....................] duration:
2.27745772s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:07:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.277481004s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
783.489958ms trace:
https://jaeger.dev.storj.io/trace/16ac22bfe521b3ea
FAIL (2026-01-28 13:06:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 783.503863ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
1.970625771s trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:06:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.970644837s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [..................X.XXXXXXXXXXXXXXXXXX.XX...................] duration:
2.281649678s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:06:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.281673724s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
6.174056832s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:06:32 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 6.174079675s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
6.39440583s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:06:15 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 6.394419084s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.001254941s trace:
https://jaeger.dev.storj.io/trace/2a302f1903a04aa4
FAIL (2026-01-28 13:06:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00135591s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001418664s trace:
https://jaeger.dev.storj.io/trace/2ec3e8fda11878e1
FAIL (2026-01-28 13:06:09 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001434454s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [...................X.XXXXXXXXXXXXXXXXXX.XX..................] duration:
2.591105595s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:05:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.591131042s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [....................X.XXXXXXXXXXXXXXXXXX.XX.................] duration:
2.24576247s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:04:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.245775955s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [.....................X.XXXXXXXXXXXXXXXXXX.XX................] duration:
2.698103486s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:04:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.698135115s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
21 out of 60 is failed (threshold=3) [......................X.XXXXXXXXXXXXXXXXXX.XX...............] duration:
2.678290873s trace:
https://jaeger.dev.storj.io/trace/7dbaa9b49c8e04d
FAIL (2026-01-28 13:03:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 23)
======LOGS======
load=271346]: Uploading piece...
Jan 28 12:51:50.184409631 (308h55m28.366854632s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184678012 (308h55m28.367123013s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.184816551 (308h55m28.367261552s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185220726 (308h55m28.367665726s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185238389 (308h55m28.367683389s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.185767077 (308h55m28.368212076s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186257063 (308h55m28.368702062s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.186750323 (308h55m28.369195324s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.187393415 (308h55m28.369838415s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Uploading piece...
Jan 28 12:51:50.298183821 (308h55m28.480628822s) [noise=true , address=185.167.204.205:4017 , piece_id=BEU3G2LERHFAI7LZMHRE6QE2FXHAXPQRDAEY7VLELIS2NTOOFRLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: write tcp 5.161.89.252:24554->185.167.204.205:4017: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.302365945 (308h55m28.484810935s) [noise=true , address=185.167.204.205:4008 , piece_id=HUJPT4F22AV6LXL7G4E5SS45QHO4Y63AKLELUYLJPST4E4H3UAWQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: write tcp 5.161.89.252:24764->185.167.204.205:4008: write: connection refused; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Jan 28 12:51:50.308007363 (308h55m28.490452363s) [noise=true , address=185.167.204.205:4020 , piece_id=E5R424DW66O6SEI2TGMZ4OA45QZHEGY4W3TYCXIF67NLJKWSKL7Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 5.161.89.252:56272->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:51:50.649323054 (308h55m28.831768055s) [upload=271346]: Next returned nil segment
Jan 28 12:51:50.649338894 (308h55m28.831783904s) [upload=271346]: Waiting for error group managing segments...
Jan 28 12:51:50.843663241 (308h55m29.02610824s) [noise=true , address=185.167.204.206:4057 , piece_id=TPMTJVSONKQCBHPARD34BWSBY6G2ESJX35DIT6KIE3KPHBGNFJIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.860728972 (308h55m29.043173951s) [noise=true , address=185.167.204.205:4035 , piece_id=YME24UO22KJTZ5U2ZNCDXKSIFL3MQPANUCNUE5SIIWDQSPQVVGNQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.865848004 (308h55m29.048292984s) [noise=true , address=89.149.201.147:4017 , piece_id=IZBI3TSLULG7Z7FQCL5PO4OD7BNAAYBR4N5NCFXNFOF5VLM32BRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.868016204 (308h55m29.050461174s) [noise=true , address=185.167.204.206:4071 , piece_id=7IH4EISSD763LQWLPJWJAWNBBD7576TW2KYRJUYN2LJ2JHMYROOQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.874433082 (308h55m29.056878081s) [noise=true , address=81.171.14.70:4046 , piece_id=NBJH47N24FLXZ4PW7P4ZFI3YSW6ZWI34S5ZIJMDBWGUVIMHK3LKA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876146613 (308h55m29.058591612s) [noise=true , address=81.171.14.70:4044 , piece_id=GYJNV7K3QKBRKQX5OLAVY2WN6HNZMPH63PDC2S4MRITWHXNLJDJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876299508 (308h55m29.058744509s) [noise=true , address=185.167.204.206:4044 , piece_id=WWIS2BOYXTEZXCUUE5M356A25LRZH33TEV2SIUCKRM2M6A3LZ2QA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.876910951 (308h55m29.059355951s) [noise=true , address=185.167.204.206:4058 , piece_id=PPNS6OSYK32URJXJBFVTWH6LPMHJUTIFES6G23XOOWU7LBIC7FIQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.878549080 (308h55m29.060994081s) [noise=true , address=81.171.14.70:4026 , piece_id=5FNKVITLJVTCWBT5UT3WBAXIKQRRZAGE3UKN4VAWC5XRXTP42ZRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.879597960 (308h55m29.06204296s) [noise=true , address=185.167.204.205:4034 , piece_id=MW6PCDOGOPUN7BBECKQD4RL74OFBWQJPMTDMOEC7CAGB73XITKKQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.880052580 (308h55m29.06249758s) [noise=true , address=185.167.204.205:4024 , piece_id=LBCC6OLXLDO3ZVU7OTXD4UFT6FQOHYB6DMHFPNXJW5TT4LQ4DTLQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.881826874 (308h55m29.064271874s) [noise=true , address=185.167.204.206:4072 , piece_id=GN3EORXBPUGH354PYGXTIWIZV4CMASAZNME2SMGKGE4GQTOUNHLA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.882524889 (308h55m29.064969888s) [noise=true , address=185.167.204.205:4023 , piece_id=LBS2USJN5CVWIHPOOKDYAVXHNDMPOXSUS6RMRFYADKTNKSNOW3DQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.883214717 (308h55m29.065659717s) [noise=true , address=185.167.204.206:4038 , piece_id=XHZCDD5PKKVVQCIVPYRWPBNXV2WRGFVISHMAMOBGQQBGIE76WKAA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.888303984 (308h55m29.070748984s) [noise=true , address=185.167.204.206:4043 , piece_id=HPRFDI3HA7WYH4PXNUH6HAFZTJZU3PZGWWQFTSTBTWXJKLNAZVMA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.899747231 (308h55m29.082192232s) [noise=true , address=185.167.204.205:4003 , piece_id=UUAU3ADTVJPMTBAZLBRTMWA4B7EG2ZK3BZLVWAVFS6Z3C6SOWVJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.906286668 (308h55m29.088731668s) [noise=true , address=89.149.201.147:4021 , piece_id=U7DQO73JQW2UUMREKJNLWTEBVNYOVF5VAIFZF3QK22FUGRLKKKGQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.907099736 (308h55m29.089544736s) [noise=true , address=81.171.14.70:4033 , piece_id=GWTT7ER762EM2VM4OAMLJKTCBVAAIXNISKHZZ5W23YGNWFN75MZA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.908491366 (308h55m29.090936366s) [noise=true , address=89.149.201.147:4023 , piece_id=JGYEFMC2CGAYTHYCJHHDQ4CDIVKOY56IZFHOKWNVFWV2KEBSL7TQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.913455309 (308h55m29.095900308s) [noise=true , address=81.171.14.70:4037 , piece_id=ZD2YZFWQAGDWRQYU27QF7K6NWZXJZFSM5N7RHEU5HFGJXQZWMDRA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:50.922826225 (308h55m29.105271226s) [noise=true , address=81.171.14.70:4043 , piece_id=BPIT5OOFKSEAHTXVKI334YKTJY7EN25QI3X4WZIOXJJWZP56F3SQ, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.362248849 (308h55m29.544693849s) [noise=true , address=89.149.201.147:4022 , piece_id=OSZMX3RF7BF7R4BRVR7UFRTNYHPFG5WAW5YBTYZZVPR5AYD5G2YA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:51:51.380533927 (308h55m29.562978937s) [noise=true , address=89.149.201.147:4006 , piece_id=YYH4E3SCRBGH7VVNXKO6FGTFEYNHYBXMWM43XF4ATS6GNQ6D3N4Q, seg_pos={0 0}, upload=271346]: Done uploading piece. err: <nil>
Jan 28 12:52:49.756393715 (308h56m27.938838695s) [noise=true , address=185.167.204.205:4010 , piece_id=6JGYL7KZGHFJDBYHX6UOSTQRTP63MTUQNRD3I36DHMONKRVTDPJA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 5.161.89.252:30290->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756442386 (308h56m27.938887355s) [noise=true , address=185.167.204.205:4012 , piece_id=WUGWJ2OZUHC5KBHFJNTBOOFLUOSSO57HG3BOQJ4HRALEELLFGUQA, seg_pos={0 0}, upload=271346]: Done uploading piece. err: ecclient: upload failed (node:1V4S6bUds1LgcrVccy5NkBJMAG3QaGFJqhsYtVboesSdH2PUFh, address:185.167.204.205:4012): protocol: read tcp 5.161.89.252:15526->185.167.204.205:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 28 12:52:49.756461832 (308h56m27.938906812s) [seg_pos={0 0}, upload=271346]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil>
Jan 28 12:52:49.861788280 (308h56m28.04423325s) [upload=271346]: Done uploading segments...
Jan 28 12:52:49.861817014 (308h56m28.044261994s) [upload=271346]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23)
Duration: 2.678315599s
core
uplink
eu-select-1
eu1-placement-15
|