Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 370.543034ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:51:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 370.572709ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 352.843169ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:51:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 352.863217ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 377.40371ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:50:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 377.423267ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 402.25191ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:50:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 402.336439ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 317.583951ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:49:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 317.602716ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 419.297511ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:48:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 419.315906ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 349.418583ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:47:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 349.446015ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 421.438663ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:47:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 421.463058ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 415.802292ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:46:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 415.817952ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 378.38313ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:46:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 378.400633ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 329.767562ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:44:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 329.786977ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 421.042734ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 421.05674ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 587.05375ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:42:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 587.078275ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 330.584758ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:41:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 330.607531ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 370.690199ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:41:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 370.709144ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 334.839646ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:40:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 334.863791ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 339.691639ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:40:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 339.710966ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 419.946055ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:39:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 419.964309ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 385.264182ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:38:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 385.286454ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 410.963211ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:37:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 410.991905ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 327.664703ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:36:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 327.685461ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 384.180559ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:35:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 384.199475ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 422.786309ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:34:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 422.807028ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 470.387415ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:34:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 470.405408ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 373.708486ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:33:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 373.732982ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 524.112649ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:32:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 524.130711ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 485.954926ms trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:32:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 485.974252ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 19.97169476s trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:31:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 19.971727641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.018604866s trace: https://jaeger.dev.storj.io/trace/6b80ce57ce9db094 FAIL (2025-12-20 23:31:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Dec 20 23:30:35.512719312 (368h1m23.210287648s) [upload=323601]: Starting upload Dec 20 23:30:35.512837753 (368h1m23.210406089s) [upload=323601]: Done starting upload Dec 20 23:30:35.513154085 (368h1m23.21072239s) [upload=323601]: Uploading segments... Dec 20 23:30:35.513309084 (368h1m23.21087739s) [upload=323601]: Got next segment. Inline: false Dec 20 23:30:35.513339551 (368h1m23.210907857s) [upload=323601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 20 23:30:35.513345202 (368h1m23.210913507s) [upload=323601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 20 23:30:35.534163623 (368h1m23.231731958s) [seg_pos={0 0}, upload=323601]: Begin upload segment... Dec 20 23:30:35.534210169 (368h1m23.231778506s) [seg_pos={0 0}, upload=323601]: Done begin upload segment. Dec 20 23:30:35.534236899 (368h1m23.231805205s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534308914 (368h1m23.23187725s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534827292 (368h1m23.232395628s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.534975829 (368h1m23.232544136s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535282222 (368h1m23.232850558s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535436871 (368h1m23.233005177s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.535700173 (368h1m23.233268519s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536245622 (368h1m23.233813958s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.536719657 (368h1m23.234287994s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537171862 (368h1m23.234740199s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537269224 (368h1m23.23483754s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.537732800 (368h1m23.235301106s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538164466 (368h1m23.235732783s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.538602725 (368h1m23.236171031s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.539013292 (368h1m23.236581598s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.540406356 (368h1m23.237974691s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Uploading piece... Dec 20 23:30:35.597304347 (368h1m23.294872683s) [noise=true , address=77.90.31.21:4877 , piece_id=4VOXYHF5BRK5SG6JQ2XRDZRMPVVQMF62AVF5Z7JLP2IWAODHPAPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 20 23:30:35.624329021 (368h1m23.321897347s) [upload=323601]: Next returned nil segment Dec 20 23:30:35.624340653 (368h1m23.321908979s) [upload=323601]: Waiting for error group managing segments... Dec 20 23:30:35.633456612 (368h1m23.331024918s) [noise=true , address=145.79.227.33:4869 , piece_id=ATJSUN336XODGAIGM3BNJTBI6WLPTLIQJOFBPP62EZZ625QSW4ZA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633500484 (368h1m23.331068791s) [noise=true , address=145.79.227.11:4732 , piece_id=ENMVKW7UVWBNUXLL6AM3G4UDHTWRPAZ7C2LYRCJWOE7YSHSKQLEQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.633855477 (368h1m23.331423783s) [noise=true , address=145.79.227.51:4411 , piece_id=IILHGKRLTCE5DJHMCR7JI2UPMJXTYLCM27QZCCIXPK43DUEF472Q, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635804339 (368h1m23.333372655s) [noise=true , address=77.90.31.16:4735 , piece_id=4MBZWIRHN4DVUNM27JJOQ7SMYFJXQFATQT4WVSNZSW24ILWQCGIQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.635843522 (368h1m23.333411828s) [noise=true , address=77.90.31.11:4554 , piece_id=7PZU3IKBD7C5HJXUNPPIQN4VZTMQWMMF5YR3U5EDWDGAPFT4FARQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.637274245 (368h1m23.334842551s) [noise=true , address=145.79.227.15:4876 , piece_id=ZGFXNTMF4AJZSNGMUSA5COEQM7L7MB5VG6SYOIHSKSI4WW5EBBPQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.644145671 (368h1m23.341713976s) [noise=true , address=145.79.227.36:4517 , piece_id=YMFMX4FNY476CPDAET5UEQULCCRM67ZRES6TTCWXOL4L6RCXURXQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.658313378 (368h1m23.355881714s) [noise=true , address=45.91.116.10:4082 , piece_id=UAP6KGDFW5TLKNTB7IVR5GD2K5XGBIWFMFV56NGSFQ6CDCNFWTHA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:30:35.687365009 (368h1m23.384933325s) [noise=true , address=145.79.227.29:4942 , piece_id=OUXK2FJNMQNUPGLQJUQSML4CISLXNH6OZCBDUFDK2DGQ3FCH5UHQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: <nil> Dec 20 23:31:05.501472952 (368h1m53.199041288s) [noise=true , address=145.79.227.40:4050 , piece_id=JUSKFHFYTJTXLEPHTEOIL5ZG2ZPYDZXI63RHFBI7LLBO42ISGMPA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501500343 (368h1m53.199068679s) [noise=true , address=145.79.227.43:4159 , piece_id=UEKE4Z56W7T4XCT4CLYI5KEZJXBTFQOT2PSFX55UWNPSQ4XDVHLQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501653199 (368h1m53.199221535s) [noise=true , address=145.79.227.42:4122 , piece_id=V4YPTMTDQT22MF2OKLOAIQIOJT452CXW2IXRSNDK4LRGQHZ4N6QQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501674789 (368h1m53.199243125s) [noise=true , address=145.79.227.30:4761 , piece_id=2W4BG4BD6XWKRWUTFASBN7CC5XNDR5I5T5AXQJSHIYOXH7VHCPCA, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501824239 (368h1m53.199392574s) [noise=true , address=45.91.116.4:4940 , piece_id=TAPD6QVN53BFLDZKJDC23PFT5LMTSA23GQSOIGEEHE6USTN5CGAQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501848043 (368h1m53.199416379s) [noise=true , address=145.79.227.32:4835 , piece_id=SJOXQEITF2VJKQFYTGSSHTAXEBVEZNWDVD4AARAHP4YWBY5MWWMQ, seg_pos={0 0}, upload=323601]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF Dec 20 23:31:05.501917102 (368h1m53.199485437s) [seg_pos={0 0}, upload=323601]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Dec 20 23:31:05.517303836 (368h1m53.214872181s) [upload=323601]: Done uploading segments... Dec 20 23:31:05.517336988 (368h1m53.214905334s) [upload=323601]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.018620744s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 525.136911ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:30:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 525.156507ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 787.549527ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:30:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 787.57291ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 475.886058ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:29:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 475.93506ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 701.844173ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:28:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 701.863028ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 571.623062ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:27:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 571.649432ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 470.164377ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:26:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.184576ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 577.94963ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:26:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 577.968324ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 482.614103ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:25:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.631225ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 728.03297ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:24:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 728.048789ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 476.136142ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:23:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 476.150176ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 775.44726ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:23:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 775.468008ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 487.586483ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:23:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.609927ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 667.378117ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:22:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 667.396952ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 472.67799ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:21:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.693671ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 488.202072ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:20:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 488.214935ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 484.455632ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:19:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.477985ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 477.35076ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:19:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.373282ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 497.105184ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:18:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 497.121333ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 553.069762ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:17:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.086975ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 479.638134ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:17:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.650316ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 583.558301ms trace: https://jaeger.dev.storj.io/trace/e0e6e3e259865f0 FAIL (2025-12-20 20:17:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 583.57915ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)