Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration:
623.476131ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:59:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 623.492732ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............X..X.X....X.X.................................] duration:
364.807579ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:59:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 364.823688ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration:
441.185507ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:59:38 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 441.205215ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
4.080828346s trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:59:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 4.0808783s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........XXX.................................................] duration:
751.031993ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:59:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 751.048264ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration:
407.259102ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:59:10 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 407.27937ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
491.27691ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:59:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 491.301996ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........XX.XX...............................................] duration:
679.392159ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:59:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 679.425893ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........XXX................................................] duration:
635.931367ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:58:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 635.960843ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration:
378.01791ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:58:50 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 378.039931ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............X..X.X....X.X................................] duration:
343.671219ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:58:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 343.686879ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration:
484.09282ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:58:34 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 484.105103ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
495.47461ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:58:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 495.494619ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........XX.XX..............................................] duration:
482.239498ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:58:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 482.263493ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X..X.X....X.X...............................] duration:
364.883606ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:57:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 364.896911ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration:
710.624483ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:57:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 710.642988ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration:
402.112729ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:57:51 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 402.132036ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
506.670705ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:57:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 506.706422ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration:
552.531652ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:57:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 552.562189ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X..X.X....X.X..............................] duration:
412.19261ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:57:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 412.211255ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
391.602246ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:57:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 391.619368ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............XXX.............................................] duration:
726.520955ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:57:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 726.537696ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..........XX.XX.............................................] duration:
472.441564ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:57:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.468245ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................X..X.X....X.X.............................] duration:
392.852642ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:57:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 392.869663ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration:
475.472045ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:57:01 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 475.494497ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration:
376.257443ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:56:37 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 376.357609ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
441.141274ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:56:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 441.183053ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................X..X.X....X.X............................] duration:
338.637217ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:56:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 338.653688ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...........XX.XX............................................] duration:
658.30526ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:56:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 658.318734ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............XXX............................................] duration:
579.492656ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:56:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 579.67701ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
377.991038ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:56:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 378.047844ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration:
457.890681ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:55:51 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 457.906282ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration:
650.938107ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:55:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 651.253396ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............XX.XX...........................................] duration:
619.479742ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:55:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 619.509678ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
437.786228ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:55:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 437.806575ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................X..X.X....X.X...........................] duration:
442.400013ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:55:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 442.428598ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration:
807.573393ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:55:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 807.585026ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
437.852382ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:55:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 437.875065ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................X..X.X....X.X..........................] duration:
407.347601ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:55:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 407.424804ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration:
423.934463ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:54:47 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 423.950523ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................XXX.........................................] duration:
553.65949ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:54:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 553.669599ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............XX.XX..........................................] duration:
501.492191ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:54:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.511677ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration:
394.77831ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:54:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 394.797416ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............XX.XX.........................................] duration:
635.444226ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:54:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 635.487456ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................X..X.X....X.X.........................] duration:
338.492848ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:54:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 338.505943ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration:
366.382234ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:54:09 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 366.402302ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
464.469771ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:53:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 464.494679ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................XXX........................................] duration:
687.628746ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:53:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 687.644856ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................X..X.X....X.X........................] duration:
394.31442ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:53:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 394.339486ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
502.040115ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:53:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 502.058569ms
core
uplink
us-select-1
|