Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration:
533.751242ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:53:24 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 533.767322ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration:
577.259616ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:53:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 577.280144ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................X..X.X....X.X.......................] duration:
345.11734ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:53:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 345.135916ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............XX.XX........................................] duration:
514.393692ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:53:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 514.409861ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
466.250338ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:52:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 466.261819ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [................XX.XX.......................................] duration:
444.345942ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:52:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 444.365417ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................XXX......................................] duration:
572.547945ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:52:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 572.586786ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration:
471.805086ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:52:18 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 471.821185ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
374.20705ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:52:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 374.229341ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........................X..X.X....X.X......................] duration:
392.949753ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:52:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 392.982685ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration:
428.683344ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:51:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 428.699985ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
451.052411ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:51:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 451.068161ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.................XX.XX......................................] duration:
669.828185ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:51:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 669.844947ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration:
585.358284ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:51:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 585.375547ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration:
411.929197ms trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:50:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 411.948703ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........................X..X.X....X.X.....................] duration:
487.946878ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:50:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 487.975861ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
565.636834ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:50:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 565.659687ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
47.837349797s trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:50:38 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 47.837362922s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration:
561.343372ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:50:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 561.359963ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..................XX.XX.....................................] duration:
447.229398ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:50:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 447.24624ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...................XX.XX....................................] duration:
497.025572ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:50:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 497.044818ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
476.402734ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:50:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 476.418694ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....................XX.XX...................................] duration:
497.073098ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:49:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 497.099236ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.020532274s trace:
https://jaeger.dev.storj.io/trace/3f8f2ad7a1b050c3
FAIL (2025-12-10 23:49:51 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
ress=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052029146 (128h19m38.749597463s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052141046 (128h19m38.749709352s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052441006 (128h19m38.750009322s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052750193 (128h19m38.750318499s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.052998327 (128h19m38.750566633s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053236983 (128h19m38.750805289s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.053475318 (128h19m38.751043634s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.054824157 (128h19m38.752392463s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055115993 (128h19m38.752684298s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.055686568 (128h19m38.753254874s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056065867 (128h19m38.753634183s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056443773 (128h19m38.754012089s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.056804797 (128h19m38.754373113s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.057906104 (128h19m38.75547442s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.058956697 (128h19m38.756525003s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.060118848 (128h19m38.757687154s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.061721401 (128h19m38.759289737s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Uploading piece...
Dec 10 23:48:51.150320094 (128h19m38.84788843s) [upload=112648]: Next returned nil segment
Dec 10 23:48:51.150332217 (128h19m38.847900543s) [upload=112648]: Waiting for error group managing segments...
Dec 10 23:48:51.160508446 (128h19m38.858076783s) [noise=true , address=45.91.116.6:4981 , piece_id=ZHIXUU24MJEHSD7AUMRZ7O3PKXADOHRZ3BFSPIK4SP4KW6RX3CAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.161744135 (128h19m38.85931247s) [noise=true , address=145.79.227.13:4828 , piece_id=DDYJAT32N5QZSM7ZYXZH4G5GN37QKSZX5BZLJZEM3NUPMPQM3VAQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.163011943 (128h19m38.860580279s) [noise=true , address=145.79.227.2:4148 , piece_id=SF7X5NMFKVOIHDIIGFQ3TOWCDADJ33TEZCR3G62WEQCEZXQB3SQA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.169606942 (128h19m38.867175278s) [noise=true , address=45.91.116.5:4320 , piece_id=Z5LZEAJH3LNFSUTMOQNN6JI5X37DOJ7ZSRR2IF5L6JJ77SDU26KQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.172780780 (128h19m38.870349096s) [noise=true , address=45.91.116.8:4077 , piece_id=OLCHS5AAF3FLZWZ3XANRU5E6WZE5IVFKOJM5YRKCKEWWYKXIDMSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.173196809 (128h19m38.870765114s) [noise=true , address=145.79.227.15:4878 , piece_id=Z2C4NACPFBLO2R4WH5QDTNCVLEEI7YXYN3NRKTMCE43FDH25I7TA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.182047871 (128h19m38.879616187s) [noise=true , address=45.91.116.9:4283 , piece_id=OVFNA3FIRHPQFKBB4WDUKM7Q762CLXP72FPKJ2RBNQGQYT7KMFMA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.185707228 (128h19m38.883275544s) [noise=true , address=145.79.227.31:4819 , piece_id=RNJ554KO2WGCVV7FUXWXNWL777U6EDNL3RIPUOD5OSOVKALHSECQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.261700924 (128h19m38.95926926s) [noise=true , address=5.231.73.3:4027 , piece_id=QHV2EBEETGGEAPBEKQINIQCBJUBJO5TKN2NI7ZYEXPDBHIHS5NBA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:51.446108900 (128h19m39.143677216s) [noise=true , address=145.79.227.12:4794 , piece_id=NXE2C5MR5VVNR5WTMD6U3E3I57BZYE6MO7K5SAA5YQ6SJM4IUOAA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:48:52.056416207 (128h19m39.753984544s) [noise=true , address=145.79.227.44:4206 , piece_id=SIXHU4SOZRVN37ESNM7ZQ3COWIXVDXHBE2NOW4PZLPY6EJWV2MTQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: <nil>
Dec 10 23:49:51.011005424 (128h20m38.70857377s) [noise=true , address=77.90.31.15:4720 , piece_id=7K642UKO2I2F4GBTUD3W2GTBKQJ24LYTLQ525H3QL6KDE3VH7RPQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 5.161.89.252:63128->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011373271 (128h20m38.708941607s) [noise=true , address=77.90.31.11:4577 , piece_id=7T3KQAL5H7WN2ZVRI6ZJPHOVPOLYE2SBXP2IFYFEE37SG6TCVJSA, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:49620->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011669915 (128h20m38.709238251s) [noise=true , address=77.90.31.10:4537 , piece_id=IIUDZTR3DISMG3GUSGAKRT23RPU4WD6JU32XFALCGB7LDV7Y2A2A, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 5.161.89.252:57812->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.011983521 (128h20m38.709551957s) [noise=true , address=77.90.31.16:4766 , piece_id=3ERY3MBSNUJGLDPOKXBM7KKATAS2EBCLDPF54T7FEIGW2PHZNW3Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: write tcp 5.161.89.252:17400->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012137407 (128h20m38.709705744s) [noise=true , address=77.90.31.13:4658 , piece_id=ZGEQ6RZS4V36U54ZJZXBBMBF6OESKD43JQD5TCC4UK4DPW2DGEYQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.012256571 (128h20m38.709824907s) [noise=true , address=77.90.31.12:4614 , piece_id=H7DHAUDVJKYCKWAEIM4KVODHXWAR3GVAS27A53BKSMTFQAF7SGOQ, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: write tcp 5.161.89.252:50544->77.90.31.12:4614: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013262780 (128h20m38.710831116s) [noise=true , address=77.90.31.17:4783 , piece_id=Y6HRVXYXNHMMMRXEUJSVTC2S7XKQHQ76HACELKBZCCPUMLAOGO6Q, seg_pos={0 0}, upload=112648]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: write tcp 5.161.89.252:14254->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:51.013498180 (128h20m38.711066527s) [seg_pos={0 0}, upload=112648]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:51.028006893 (128h20m38.725575239s) [upload=112648]: Done uploading segments...
Dec 10 23:49:51.028041437 (128h20m38.725609774s) [upload=112648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.02054652s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................X..X.X....X.X....................] duration:
319.050098ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:49:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 319.072099ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................XXX...................................] duration:
587.71029ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:49:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 587.739274ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
365.642233ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:49:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 365.671938ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....................XX.XX..................................] duration:
475.108558ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:49:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 475.142932ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................X..X.X....X.X...................] duration:
383.078632ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:48:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 383.092899ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
520.432233ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:48:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 520.448662ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.017368607s trace:
https://jaeger.dev.storj.io/trace/2d1af6c09e6f7378
FAIL (2025-12-10 23:48:50 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
42YWZYXCM2EVZAS22INB6SKEPWJZCUB6P44A3S5TKIA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.987959492 (128h18m38.685527828s) [noise=true , address=45.91.116.4:4960 , piece_id=IPO5QKLJ7N3HHRWZMY4VR7TCXNH5GNPIAUU4ZI6VMHY7CBGW7YIA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.988235327 (128h18m38.685803644s) [noise=true , address=145.79.227.31:4824 , piece_id=Y7SV7SVNYAKMIIUWQJGL3YM5EOP6BVJN3OFJF5FY2AMHXW4Y7PLA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.988482358 (128h18m38.686050675s) [noise=true , address=45.91.116.7:4023 , piece_id=V3XGQQER7VWP6Z77L3PAOGXFHN5GVPY4MRDY4UV4Z3BCQG2FG3UA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.988908064 (128h18m38.68647638s) [noise=true , address=77.90.31.12:4605 , piece_id=RQP4HT2IRRJ3VGSPYMWXXNM36G3WS2ZXEG7RPOLEHIXY7CXJZO3A, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.989327427 (128h18m38.686895733s) [noise=true , address=77.90.31.16:4767 , piece_id=LODDBU4CSWCZC7LVISJYDVCIOSWUNZ7XMMKPDH56E3VTH2IORZIQ, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.989613271 (128h18m38.687181577s) [noise=true , address=77.90.31.21:4882 , piece_id=BJVGFZ2PZWUWJT5XKPYG6DAIHFXWFQ4TGAOKCLP4E5JYB62PHPZA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.990000044 (128h18m38.68756835s) [noise=true , address=77.90.31.18:4828 , piece_id=RND53PTUK3CFEAQEE6YWCICWQK7EYB6MODHN2NQZB6I7JUPUEFHA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.990425709 (128h18m38.687994026s) [noise=true , address=45.91.116.7:4042 , piece_id=SUJGI2JMYDKDVJLKIRFHH3NJL52HMKHK77SMGE5ZH25BBCAC4W7Q, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.991955157 (128h18m38.689523473s) [noise=true , address=45.91.116.5:4317 , piece_id=4ANZXXTFKA6OVBQE3BMJJCH7KNS22CA3CHDNLQJGOLIOUML7V2VQ, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.993233764 (128h18m38.69080208s) [noise=true , address=45.91.116.9:4286 , piece_id=BM6BPU26SSJV7EHW6264EY72Z5ZLWA7DVE26LAERV2HSOPFB2DNA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.994267094 (128h18m38.69183541s) [noise=true , address=45.91.116.12:4254 , piece_id=JS5EXDY426XV6BHJSY7LXNHSWKKOGFAIFCW2LNJ25UMHCGQ3I53Q, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.995295917 (128h18m38.692864233s) [noise=true , address=145.79.227.6:4282 , piece_id=OB3JKBWRGM3CZC4IVIGJPDTERCXXWQKI2D5WHG7UURAGH7X4ND7Q, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.996448809 (128h18m38.694017116s) [noise=true , address=145.79.227.30:4772 , piece_id=X5NBD6EUAGIZHF4P2BATPZZXDEWKBA34WDGAO4BTGDHUGNVY5P5A, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.997833177 (128h18m38.695401493s) [noise=true , address=77.90.31.15:4731 , piece_id=GXRVOM74IIOUA4NWZLDVNGH3UYXRWQO3PYFADEXKBTSOVQ3CNHIA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:50.999204518 (128h18m38.696772834s) [noise=true , address=77.90.31.17:4773 , piece_id=675XQIJSYBME7OBCBORB7VUIQMCQRXQNJJMDDJW6BBZABMOFQNJA, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:51.000449273 (128h18m38.698017599s) [noise=true , address=77.90.31.18:4812 , piece_id=6ITVGX4L5A3544BD6LKYINDNPCEW53ZDS4JHNLX3JSR4BCDZUJLQ, seg_pos={0 0}, upload=112634]: Uploading piece...
Dec 10 23:47:51.054032527 (128h18m38.751600843s) [upload=112634]: Next returned nil segment
Dec 10 23:47:51.054041954 (128h18m38.75161027s) [upload=112634]: Waiting for error group managing segments...
Dec 10 23:47:51.061954214 (128h18m38.759522521s) [noise=true , address=45.91.116.7:4042 , piece_id=SUJGI2JMYDKDVJLKIRFHH3NJL52HMKHK77SMGE5ZH25BBCAC4W7Q, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:47:51.062856840 (128h18m38.760425156s) [noise=true , address=45.91.116.9:4286 , piece_id=BM6BPU26SSJV7EHW6264EY72Z5ZLWA7DVE26LAERV2HSOPFB2DNA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:47:51.063328973 (128h18m38.760897289s) [noise=true , address=45.91.116.12:4254 , piece_id=JS5EXDY426XV6BHJSY7LXNHSWKKOGFAIFCW2LNJ25UMHCGQ3I53Q, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:47:51.074349298 (128h18m38.771917615s) [noise=true , address=45.91.116.4:4960 , piece_id=IPO5QKLJ7N3HHRWZMY4VR7TCXNH5GNPIAUU4ZI6VMHY7CBGW7YIA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:47:51.081326602 (128h18m38.778894918s) [noise=true , address=45.91.116.7:4023 , piece_id=V3XGQQER7VWP6Z77L3PAOGXFHN5GVPY4MRDY4UV4Z3BCQG2FG3UA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:47:51.081994209 (128h18m38.779562525s) [noise=true , address=45.91.116.5:4317 , piece_id=4ANZXXTFKA6OVBQE3BMJJCH7KNS22CA3CHDNLQJGOLIOUML7V2VQ, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:47:51.085606327 (128h18m38.783174643s) [noise=true , address=145.79.227.31:4824 , piece_id=Y7SV7SVNYAKMIIUWQJGL3YM5EOP6BVJN3OFJF5FY2AMHXW4Y7PLA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:47:51.089703280 (128h18m38.787271596s) [noise=true , address=145.79.227.6:4282 , piece_id=OB3JKBWRGM3CZC4IVIGJPDTERCXXWQKI2D5WHG7UURAGH7X4ND7Q, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:47:52.051505314 (128h18m39.74907363s) [noise=true , address=145.79.227.30:4772 , piece_id=X5NBD6EUAGIZHF4P2BATPZZXDEWKBA34WDGAO4BTGDHUGNVY5P5A, seg_pos={0 0}, upload=112634]: Done uploading piece. err: <nil>
Dec 10 23:48:50.946688279 (128h19m38.644256585s) [noise=true , address=77.90.31.13:4658 , piece_id=V7Y7A2DZ5UOTYNQVA6XRH4W6JHJ4XYZRF2GHZYXJ5WRIPXQZSAXQ, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.946773227 (128h19m38.644341533s) [noise=true , address=77.90.31.18:4812 , piece_id=6ITVGX4L5A3544BD6LKYINDNPCEW53ZDS4JHNLX3JSR4BCDZUJLQ, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.946832819 (128h19m38.644401135s) [noise=true , address=77.90.31.21:4882 , piece_id=BJVGFZ2PZWUWJT5XKPYG6DAIHFXWFQ4TGAOKCLP4E5JYB62PHPZA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: rpc: dial tcp 77.90.31.21:4882: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.946939548 (128h19m38.644507854s) [noise=true , address=77.90.31.15:4731 , piece_id=GXRVOM74IIOUA4NWZLDVNGH3UYXRWQO3PYFADEXKBTSOVQ3CNHIA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo, address:77.90.31.15:4731): protocol: rpc: dial tcp 77.90.31.15:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.946969454 (128h19m38.644537771s) [noise=true , address=77.90.31.17:4773 , piece_id=675XQIJSYBME7OBCBORB7VUIQMCQRXQNJJMDDJW6BBZABMOFQNJA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.947029786 (128h19m38.644598102s) [noise=true , address=77.90.31.16:4767 , piece_id=LODDBU4CSWCZC7LVISJYDVCIOSWUNZ7XMMKPDH56E3VTH2IORZIQ, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: write tcp 5.161.89.252:35220->77.90.31.16:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.947104467 (128h19m38.644672783s) [noise=true , address=77.90.31.19:4845 , piece_id=EFLRPFDGC42YWZYXCM2EVZAS22INB6SKEPWJZCUB6P44A3S5TKIA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: rpc: dial tcp 77.90.31.19:4845: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.947230281 (128h19m38.644798587s) [noise=true , address=77.90.31.12:4605 , piece_id=RQP4HT2IRRJ3VGSPYMWXXNM36G3WS2ZXEG7RPOLEHIXY7CXJZO3A, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: rpc: dial tcp 77.90.31.12:4605: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.947352459 (128h19m38.644920766s) [noise=true , address=77.90.31.18:4828 , piece_id=RND53PTUK3CFEAQEE6YWCICWQK7EYB6MODHN2NQZB6I7JUPUEFHA, seg_pos={0 0}, upload=112634]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: rpc: dial tcp 77.90.31.18:4828: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:50.947369522 (128h19m38.644937838s) [seg_pos={0 0}, upload=112634]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Dec 10 23:48:50.962385711 (128h19m38.659954027s) [upload=112634]: Done uploading segments...
Dec 10 23:48:50.962399197 (128h19m38.659967502s) [upload=112634]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.01738091s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration:
663.437717ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:48:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 663.448747ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......................XX.XX.................................] duration:
461.448532ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:48:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 461.468149ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................XXX.................................] duration:
582.190224ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:48:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 582.224538ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............................X..X.X....X.X..................] duration:
329.686396ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:48:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 329.702166ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
546.894663ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:48:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 546.915531ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................XXX................................] duration:
593.756784ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:47:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 593.775196ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.015532223s trace:
https://jaeger.dev.storj.io/trace/16e602858e14a62f
FAIL (2025-12-10 23:47:50 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
pload=112621]: Got next segment. Inline: false
Dec 10 23:46:50.916500158 (128h17m38.614068474s) [upload=112621]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:46:50.916503754 (128h17m38.61407207s) [upload=112621]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:46:50.944078663 (128h17m38.641646969s) [seg_pos={0 0}, upload=112621]: Begin upload segment...
Dec 10 23:46:50.944116163 (128h17m38.64168447s) [seg_pos={0 0}, upload=112621]: Done begin upload segment.
Dec 10 23:46:50.944139056 (128h17m38.641707372s) [noise=true , address=64.49.13.140:4508 , piece_id=P22ZEQXD5R2GKB67BYJSXJ34TS3ITEJOG7ZRYXQBSTX7W3BDSUOA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.944183710 (128h17m38.641752046s) [noise=true , address=77.90.31.12:4618 , piece_id=4JBD3KL2MGXPQE22A5W5WNEN4AE7RBI64JXPM2TJGETRZOY4XKCQ, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.945063393 (128h17m38.642631738s) [noise=true , address=77.90.31.21:4910 , piece_id=72QDADE6K2O4Y3LBVBN67DHXZBKRDN2C24O2Y3GSUGT4U62LSSRA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.945652723 (128h17m38.64322106s) [noise=true , address=45.91.116.12:4256 , piece_id=3B7YB4OLR4J7MOBRXZAVG7OULDZ36QD6W2NZX3MMBP2N4OSY65VQ, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.946257443 (128h17m38.64382578s) [noise=true , address=64.49.13.138:4489 , piece_id=R2LOJDIMUKM47NGPSJLMG6PS4RZVH6U66ARYELRU6MYX5QHCNEGA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.946984762 (128h17m38.644553099s) [noise=true , address=45.91.116.9:4285 , piece_id=4JP2N4XDK7ETLZHB5SAF6TF7XKJS3INUEY5HINWFSKMNJNHY2AYA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.947594872 (128h17m38.645163209s) [noise=true , address=45.91.116.11:4149 , piece_id=CNFLMUQE3TDOATBATWGAEY6MN5QOWO2TJS4ZK43WJN7UML7VZ4KQ, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.948231241 (128h17m38.645799587s) [noise=true , address=145.79.227.41:4113 , piece_id=6K6CEQA27ZXGQHEJMIOUKPFAUCBKJMBD7ZBB6OMHTZXWKFMWOL3Q, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.948879111 (128h17m38.646447448s) [noise=true , address=5.231.73.5:4031 , piece_id=KCLS37ERXFAZLGJE62Z34IB2CQXGPIS3LVZRDZ4QX33PHS7CL5YQ, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.950015534 (128h17m38.64758384s) [noise=true , address=45.91.116.4:4966 , piece_id=RD3O3POR2QDK66OSD3X6UXAW3W7NFS4ILMN4RN66W3PEIUNEDTOA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.952580415 (128h17m38.650148721s) [noise=true , address=145.79.227.24:4598 , piece_id=6CD4WKUAWUNY3SIIWDSHJDVSN24YVNEV6JSE4TYHM7AI7XTEAFRQ, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.954932630 (128h17m38.652500946s) [noise=true , address=45.91.116.8:4061 , piece_id=EBA4BGJUIAIWLY6F7JKINMMRXFDCVZ3PSP5RAFJMK3ESMF4YCYIQ, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.958860697 (128h17m38.656429003s) [noise=true , address=145.79.227.42:4137 , piece_id=CXMQ5SDHZLOHIXYP5JQZZIHL3AQFA6VBBDMBPOYRU23EAH5O4PQQ, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.960598402 (128h17m38.658166708s) [noise=true , address=77.90.31.11:4573 , piece_id=IF75UZ73VEDI2DGU7DFEB57SPGY47FILCSEVYUPOZ54TCLGV7GKA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.962801197 (128h17m38.660369513s) [noise=true , address=77.90.31.19:4856 , piece_id=XPOWGWZL26P3DUFYRSJKTSMX4NZIY4WVQZVIUWQCPQIKRGJV6EOA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.968179023 (128h17m38.665747359s) [noise=true , address=77.90.31.16:4751 , piece_id=QO3GKPNBVMIPJR227O4VVRSSTYJZPO7RRDWEWM3PNPRNS7MAUYIQ, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.969570182 (128h17m38.667138488s) [noise=true , address=64.49.13.169:4705 , piece_id=4N476SZMO6CMPII3B55MZWBI5KLLE7K5RENOQHCBDTERS46HIOEA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:50.970740769 (128h17m38.668309085s) [noise=true , address=45.91.116.13:4162 , piece_id=DBOPEBWBQ5GPZ2BUT2EFJNGZWXDDIK6FLZU3AZZ6X2DTQVP6FOHA, seg_pos={0 0}, upload=112621]: Uploading piece...
Dec 10 23:46:51.045569519 (128h17m38.743137835s) [upload=112621]: Next returned nil segment
Dec 10 23:46:51.045606909 (128h17m38.743175225s) [upload=112621]: Waiting for error group managing segments...
Dec 10 23:46:51.056609492 (128h17m38.754177828s) [noise=true , address=45.91.116.8:4061 , piece_id=EBA4BGJUIAIWLY6F7JKINMMRXFDCVZ3PSP5RAFJMK3ESMF4YCYIQ, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.058452335 (128h17m38.756020671s) [noise=true , address=45.91.116.11:4149 , piece_id=CNFLMUQE3TDOATBATWGAEY6MN5QOWO2TJS4ZK43WJN7UML7VZ4KQ, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.064883428 (128h17m38.762451734s) [noise=true , address=145.79.227.41:4113 , piece_id=6K6CEQA27ZXGQHEJMIOUKPFAUCBKJMBD7ZBB6OMHTZXWKFMWOL3Q, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.069010918 (128h17m38.766579255s) [noise=true , address=45.91.116.13:4162 , piece_id=DBOPEBWBQ5GPZ2BUT2EFJNGZWXDDIK6FLZU3AZZ6X2DTQVP6FOHA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.070461578 (128h17m38.768029884s) [noise=true , address=45.91.116.12:4256 , piece_id=3B7YB4OLR4J7MOBRXZAVG7OULDZ36QD6W2NZX3MMBP2N4OSY65VQ, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.070974036 (128h17m38.768542372s) [noise=true , address=45.91.116.4:4966 , piece_id=RD3O3POR2QDK66OSD3X6UXAW3W7NFS4ILMN4RN66W3PEIUNEDTOA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.072575357 (128h17m38.770143664s) [noise=true , address=45.91.116.9:4285 , piece_id=4JP2N4XDK7ETLZHB5SAF6TF7XKJS3INUEY5HINWFSKMNJNHY2AYA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.095079945 (128h17m38.792648281s) [noise=true , address=145.79.227.42:4137 , piece_id=CXMQ5SDHZLOHIXYP5JQZZIHL3AQFA6VBBDMBPOYRU23EAH5O4PQQ, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.095866134 (128h17m38.79343445s) [noise=true , address=145.79.227.24:4598 , piece_id=6CD4WKUAWUNY3SIIWDSHJDVSN24YVNEV6JSE4TYHM7AI7XTEAFRQ, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.183772754 (128h17m38.88134109s) [noise=true , address=5.231.73.5:4031 , piece_id=KCLS37ERXFAZLGJE62Z34IB2CQXGPIS3LVZRDZ4QX33PHS7CL5YQ, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.548489978 (128h17m39.246058294s) [noise=true , address=64.49.13.140:4508 , piece_id=P22ZEQXD5R2GKB67BYJSXJ34TS3ITEJOG7ZRYXQBSTX7W3BDSUOA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:51.583724577 (128h17m39.281292923s) [noise=true , address=64.49.13.138:4489 , piece_id=R2LOJDIMUKM47NGPSJLMG6PS4RZVH6U66ARYELRU6MYX5QHCNEGA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:46:56.704564784 (128h17m44.40213309s) [noise=true , address=64.49.13.169:4705 , piece_id=4N476SZMO6CMPII3B55MZWBI5KLLE7K5RENOQHCBDTERS46HIOEA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: <nil>
Dec 10 23:47:50.885744244 (128h18m38.58331255s) [noise=true , address=77.90.31.12:4618 , piece_id=4JBD3KL2MGXPQE22A5W5WNEN4AE7RBI64JXPM2TJGETRZOY4XKCQ, seg_pos={0 0}, upload=112621]: Done uploading piece. err: ecclient: upload failed (node:1XuRKaSZC9qb9M17ctkcNkf5dSkBYpDBHgxMS56wThuw8s2R6, address:77.90.31.12:4618): protocol: write tcp 5.161.89.252:54590->77.90.31.12:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:50.885761867 (128h18m38.583330183s) [noise=true , address=77.90.31.11:4573 , piece_id=IF75UZ73VEDI2DGU7DFEB57SPGY47FILCSEVYUPOZ54TCLGV7GKA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: ecclient: upload failed (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV, address:77.90.31.11:4573): protocol: rpc: dial tcp 77.90.31.11:4573: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:50.885845743 (128h18m38.583414049s) [noise=true , address=77.90.31.21:4910 , piece_id=72QDADE6K2O4Y3LBVBN67DHXZBKRDN2C24O2Y3GSUGT4U62LSSRA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: write tcp 5.161.89.252:34664->77.90.31.21:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:50.885927316 (128h18m38.583495622s) [noise=true , address=77.90.31.19:4856 , piece_id=XPOWGWZL26P3DUFYRSJKTSMX4NZIY4WVQZVIUWQCPQIKRGJV6EOA, seg_pos={0 0}, upload=112621]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:50.885974434 (128h18m38.583542741s) [noise=true , address=77.90.31.16:4751 , piece_id=QO3GKPNBVMIPJR227O4VVRSSTYJZPO7RRDWEWM3PNPRNS7MAUYIQ, seg_pos={0 0}, upload=112621]: Done uploading piece. err: ecclient: upload failed (node:12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, address:77.90.31.16:4751): protocol: write tcp 5.161.89.252:52102->77.90.31.16:4751: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:50.885992166 (128h18m38.583560472s) [seg_pos={0 0}, upload=112621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 10 23:47:50.899235315 (128h18m38.596803631s) [upload=112621]: Done uploading segments...
Dec 10 23:47:50.899251966 (128h18m38.596820282s) [upload=112621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.015542932s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
396.646099ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:47:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 396.66259ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......................XX.XX................................] duration:
465.739132ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:47:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 465.761725ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............................X..X.X....X.X.................] duration:
355.589312ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:47:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 355.606765ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration:
552.366315ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:47:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 552.387204ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
478.581933ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:46:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 478.598143ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.017312947s trace:
https://jaeger.dev.storj.io/trace/7d8265e32276e3a0
FAIL (2025-12-10 23:46:50 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
d=AKCOWCJPD3Q3NVW3LVO3DMX45U2BGELO5UVU6V2ABUOOH4OIPLVQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.854710167 (128h16m38.552278472s) [noise=true , address=77.90.31.15:4713 , piece_id=GHW2MSKU6FRJLQ35ZA7D55XYLAEHBGGJPC6FMSNKHTLWZPT7FBFQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.855103431 (128h16m38.552671737s) [noise=true , address=77.90.31.16:4750 , piece_id=HBCUWKKQXYCHXGPX5FMGT7CGW35E6IXP35LEXHZG5TJCH3QG6I2A, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.855385518 (128h16m38.552953824s) [noise=true , address=77.90.31.13:4630 , piece_id=XL6PCFZEGKWTVKFQ3WFLOP2HF4R4H4OYX33HCUUD5GAXB6SVIMAQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.855709955 (128h16m38.553278271s) [noise=true , address=77.90.31.11:4566 , piece_id=5NOE75WAAU6OA4WK742OXWWS6ARCMENEJVXZ2ARE4ADQEHATVKXQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.855991530 (128h16m38.553559847s) [noise=true , address=5.231.73.2:4022 , piece_id=5HBHPLDTVVXJBCM4DBPOLSHJDL5HDS5LWMP427RFUFQRRNAUB2SQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.856264801 (128h16m38.553833117s) [noise=true , address=145.79.227.16:4931 , piece_id=F4WJR5VAUGEMHXFOLV3KCHHZNSTFCY73YAHLUN62HRBADDS6KYPQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.856564781 (128h16m38.554133087s) [noise=true , address=77.90.31.19:4871 , piece_id=4OA2RDEVD2IVGGW5BLH5O6B65EC465RDBVYWRBLTUYICC347NS6A, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.856940402 (128h16m38.554508709s) [noise=true , address=145.79.227.33:4899 , piece_id=QG7QQ4LRDJ7GGIIXEVUJZ2DBHHF3NXZ2UFGNTA2YW3O36PKFUZPQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.857288472 (128h16m38.554856788s) [noise=true , address=45.91.116.6:4999 , piece_id=5AUY4MTQCTRJO4UQ2UKORGIHX4XFL35RDCIJ7KN5MO23D5UUK5GQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.858984620 (128h16m38.556552927s) [noise=true , address=64.49.13.26:4955 , piece_id=LP34MTF7WEE6FYG23IPTC5DDKER7UIB3FDGY66UGJH3GJSALWSIA, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.859387443 (128h16m38.556955749s) [noise=true , address=145.79.227.34:4916 , piece_id=4RKEIU6IOXGXR5GQ4H33DFD4T35UOEHXB75HEBMW3PBS3KXWSTBQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.859745522 (128h16m38.557313838s) [noise=true , address=77.90.31.21:4903 , piece_id=MBHPJRXURBVR5V2O7MNQJS7UNWEQEABQZOE2ZGUNETLWW5QPJPCA, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.861100233 (128h16m38.558668539s) [noise=true , address=77.90.31.17:4789 , piece_id=7FUPUDQTNXALU5U6W62RGTI5MIQSLM4DR7HLWKHPK57KZAEBZCDQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.862450125 (128h16m38.560018441s) [noise=true , address=145.79.227.23:4564 , piece_id=AC2UHXTKFCH77BDHBWNQUZU4YTPQJ7FV4Q5YINCS4IJAVFHEWKJQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.863777083 (128h16m38.561345389s) [noise=true , address=45.91.116.9:4265 , piece_id=F44L3SP2DCZTQSFPXY4KXYBPNUOJLTRKIDZPIC2NOF5BXWOW42MQ, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.864987034 (128h16m38.56255534s) [noise=true , address=145.79.227.9:4382 , piece_id=2EJXVQ3UDSM5Z67ACOAQ442JFY625KS2KA5DMCGUSS42OQSWEWPA, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.866169933 (128h16m38.563738249s) [noise=true , address=145.79.227.16:4919 , piece_id=V67OXFGKEV3WJN67IJMYBXJ5JJXYR62ZAA3KJCW5XFDYR5UXZO5Q, seg_pos={0 0}, upload=112605]: Uploading piece...
Dec 10 23:45:50.932965297 (128h16m38.630533613s) [upload=112605]: Next returned nil segment
Dec 10 23:45:50.932974344 (128h16m38.63054265s) [upload=112605]: Waiting for error group managing segments...
Dec 10 23:45:50.942306346 (128h16m38.639874653s) [noise=true , address=145.79.227.9:4382 , piece_id=2EJXVQ3UDSM5Z67ACOAQ442JFY625KS2KA5DMCGUSS42OQSWEWPA, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:50.943030529 (128h16m38.640598836s) [noise=true , address=45.91.116.6:4999 , piece_id=5AUY4MTQCTRJO4UQ2UKORGIHX4XFL35RDCIJ7KN5MO23D5UUK5GQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:50.943784328 (128h16m38.641352634s) [noise=true , address=145.79.227.23:4564 , piece_id=AC2UHXTKFCH77BDHBWNQUZU4YTPQJ7FV4Q5YINCS4IJAVFHEWKJQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:50.944396221 (128h16m38.641964526s) [noise=true , address=145.79.227.34:4916 , piece_id=4RKEIU6IOXGXR5GQ4H33DFD4T35UOEHXB75HEBMW3PBS3KXWSTBQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:50.944815633 (128h16m38.64238394s) [noise=true , address=145.79.227.33:4899 , piece_id=QG7QQ4LRDJ7GGIIXEVUJZ2DBHHF3NXZ2UFGNTA2YW3O36PKFUZPQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:51.061955102 (128h16m38.759523408s) [noise=true , address=5.231.73.2:4022 , piece_id=5HBHPLDTVVXJBCM4DBPOLSHJDL5HDS5LWMP427RFUFQRRNAUB2SQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:51.213735849 (128h16m38.911304165s) [noise=true , address=45.91.116.9:4265 , piece_id=F44L3SP2DCZTQSFPXY4KXYBPNUOJLTRKIDZPIC2NOF5BXWOW42MQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:51.533385517 (128h16m39.230953853s) [noise=true , address=64.49.13.26:4955 , piece_id=LP34MTF7WEE6FYG23IPTC5DDKER7UIB3FDGY66UGJH3GJSALWSIA, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:51.852144343 (128h16m39.549712679s) [noise=true , address=145.79.227.16:4931 , piece_id=F4WJR5VAUGEMHXFOLV3KCHHZNSTFCY73YAHLUN62HRBADDS6KYPQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:45:51.992495860 (128h16m39.690064166s) [noise=true , address=145.79.227.16:4919 , piece_id=V67OXFGKEV3WJN67IJMYBXJ5JJXYR62ZAA3KJCW5XFDYR5UXZO5Q, seg_pos={0 0}, upload=112605]: Done uploading piece. err: <nil>
Dec 10 23:46:50.819250444 (128h17m38.51681875s) [noise=true , address=77.90.31.19:4871 , piece_id=4OA2RDEVD2IVGGW5BLH5O6B65EC465RDBVYWRBLTUYICC347NS6A, seg_pos={0 0}, upload=112605]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:50.819268557 (128h17m38.516836894s) [noise=true , address=77.90.31.16:4750 , piece_id=HBCUWKKQXYCHXGPX5FMGT7CGW35E6IXP35LEXHZG5TJCH3QG6I2A, seg_pos={0 0}, upload=112605]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:50.819459273 (128h17m38.517027579s) [noise=true , address=77.90.31.17:4789 , piece_id=7FUPUDQTNXALU5U6W62RGTI5MIQSLM4DR7HLWKHPK57KZAEBZCDQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: rpc: dial tcp 77.90.31.17:4789: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:50.819627358 (128h17m38.517195663s) [noise=true , address=77.90.31.11:4566 , piece_id=5NOE75WAAU6OA4WK742OXWWS6ARCMENEJVXZ2ARE4ADQEHATVKXQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: ecclient: upload failed (node:12wYPWKdwDaGih8VSexUfcPyZHEoCMt1VLa3XMXvn9BZtKo7dfk, address:77.90.31.11:4566): protocol: rpc: dial tcp 77.90.31.11:4566: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:50.819709552 (128h17m38.517277888s) [noise=true , address=77.90.31.13:4630 , piece_id=XL6PCFZEGKWTVKFQ3WFLOP2HF4R4H4OYX33HCUUD5GAXB6SVIMAQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:50.819796343 (128h17m38.517364659s) [noise=true , address=77.90.31.15:4713 , piece_id=GHW2MSKU6FRJLQ35ZA7D55XYLAEHBGGJPC6FMSNKHTLWZPT7FBFQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:50.819972182 (128h17m38.517540498s) [noise=true , address=77.90.31.21:4903 , piece_id=MBHPJRXURBVR5V2O7MNQJS7UNWEQEABQZOE2ZGUNETLWW5QPJPCA, seg_pos={0 0}, upload=112605]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 5.161.89.252:34660->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:50.820041982 (128h17m38.517610328s) [noise=true , address=77.90.31.14:4673 , piece_id=AKCOWCJPD3Q3NVW3LVO3DMX45U2BGELO5UVU6V2ABUOOH4OIPLVQ, seg_pos={0 0}, upload=112605]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:50.820079302 (128h17m38.517647638s) [seg_pos={0 0}, upload=112605]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:46:50.834418748 (128h17m38.531987094s) [upload=112605]: Done uploading segments...
Dec 10 23:46:50.834445628 (128h17m38.532013975s) [upload=112605]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.017324349s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............................X..X.X....X.X................] duration:
450.173928ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:46:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 450.192292ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
404.403089ms trace:
https://jaeger.dev.storj.io/trace/64ce9b418111a1dc
FAIL (2025-12-10 23:46:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload
Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload
Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments...
Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false
Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment...
Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment.
Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece...
Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment
Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments...
Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil>
Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments...
Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 404.423146ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........................XX.XX...............................] duration:
478.164566ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:46:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 478.183361ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration:
555.894353ms trace:
https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf
FAIL (2025-12-10 23:46:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 555.920051ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................X..X.X....X.X...............] duration:
359.963896ms trace:
https://jaeger.dev.storj.io/trace/1b146e92499c96d8
FAIL (2025-12-10 23:46:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 359.985045ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........................XX.XX..............................] duration:
445.803946ms trace:
https://jaeger.dev.storj.io/trace/1b53f25cc0b11955
FAIL (2025-12-10 23:45:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 445.824825ms
core
uplink
us-select-1
us1-placement-20
|