Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
413.790803ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:07:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 413.81176ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X..X.X....X.X............................................] duration:
362.58141ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:07:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 362.604674ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXX..........................] duration:
477.930519ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:07:00 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 477.941881ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.XX.........................................................] duration:
484.453174ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:06:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 484.475066ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
382.987109ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:06:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 383.005083ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X..X.X....X.X...........................................] duration:
373.830583ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:06:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 373.846153ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXX.........................] duration:
406.449757ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:06:32 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 406.466538ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [X.XX........................................................] duration:
472.837067ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:05:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.855852ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
436.89926ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:05:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 436.909841ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [XX.XX.......................................................] duration:
460.788198ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:05:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 460.801723ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X..X.X....X.X..........................................] duration:
336.809848ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:05:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 336.826389ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
440.189949ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:05:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 440.207564ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
797.259094ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:05:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 797.281106ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXX........................] duration:
449.734972ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:05:18 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 449.751663ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
599.306079ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:04:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 599.330064ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.XX.XX......................................................] duration:
493.072435ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:04:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 493.097113ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X..X.X....X.X.........................................] duration:
360.785363ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:04:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 360.80482ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
528.557145ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:04:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 528.573394ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXX.......................] duration:
459.630034ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:04:26 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 459.650102ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X..X.X....X.X........................................] duration:
368.289374ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:04:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 368.309421ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..XX.XX.....................................................] duration:
479.918683ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:04:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 479.93366ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
626.540144ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:04:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 626.557937ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXX......................] duration:
406.116055ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:03:48 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 406.132686ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X..X.X....X.X.......................................] duration:
382.785924ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:03:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 382.84805ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...XX.XX....................................................] duration:
479.471772ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:03:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 479.53549ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
350.916408ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:03:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 350.933499ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXX.....................] duration:
475.61303ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:03:30 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 475.630874ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.XXX........................................................] duration:
600.122323ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:03:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 600.157149ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
442.436033ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:02:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 442.45535ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X..X.X....X.X......................................] duration:
383.608639ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:02:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 383.621032ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....XX.XX...................................................] duration:
500.743999ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:02:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 500.76026ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXX....................] duration:
451.096412ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:02:41 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 451.120728ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
386.276675ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:02:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 386.295861ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration:
614.807066ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:02:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 614.828836ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........X..X.X....X.X.....................................] duration:
338.611675ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:02:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 338.661157ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...XXX......................................................] duration:
633.217357ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:02:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 633.235099ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X..X.X....X.X....................................] duration:
332.07968ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:01:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 332.095971ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXX...................] duration:
525.855958ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:01:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 525.964411ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....XX.XX..................................................] duration:
508.866811ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:01:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 508.88124ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
422.286062ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:01:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 422.307323ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration:
597.359031ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:01:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 597.372336ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXX..................] duration:
361.926556ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:01:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 361.944971ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration:
549.214383ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:01:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 549.305674ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X..X.X....X.X...................................] duration:
375.646135ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:01:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 375.692291ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......XX.XX.................................................] duration:
494.115609ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-11 00:00:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 494.131988ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......XXX...................................................] duration:
646.936134ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-11 00:00:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 646.94972ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
430.195538ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-11 00:00:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 430.213582ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXX.................] duration:
476.812964ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-11 00:00:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 476.83214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............X..X.X....X.X..................................] duration:
563.476806ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-11 00:00:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 563.498486ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......XX.XX................................................] duration:
649.061565ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:59:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 649.082525ms
core
uplink
us-select-1
us1-placement-20
|