Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 109921 reports (Page 91 of 2199)
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXX.X..............]
duration: 
605.505647ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:18:01 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 605.524032ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.............................................X..............]
duration: 
388.940624ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:17:54 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 388.958478ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [..................XXXXXXXXXXXXXXXXXXXXXXXXX.................]
duration: 
461.412246ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:17:52 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 461.424779ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..............................................X.............]
duration: 
438.139052ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:17:15 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 438.16507ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXX.X.............]
duration: 
596.72705ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:17:05 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 596.805476ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [...................XXXXXXXXXXXXXXXXXXXXXXXXX................]
duration: 
715.640374ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:16:45 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 715.678986ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [....................XXXXXXXXXXXXXXXXXXXXXXXXX...............]
duration: 
498.103397ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:16:18 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 498.121702ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...............................................X............]
duration: 
587.55793ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:16:02 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 587.577616ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [.....................XXXXXXXXXXXXXXXXXXXXXXXXX..............]
duration: 
581.643712ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:15:53 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 581.663609ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXX.X............]
duration: 
684.979537ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:15:51 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 685.001928ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [......................XXXXXXXXXXXXXXXXXXXXXXXXX.............]
duration: 
669.464917ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:15:02 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 669.503857ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [................................................X...........]
duration: 
398.938844ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:14:55 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 398.957798ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXX.X...........]
duration: 
665.886851ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:14:40 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 665.900417ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.................................................X..........]
duration: 
376.565317ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:14:06 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 376.583732ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [.......................XXXXXXXXXXXXXXXXXXXXXXXXX............]
duration: 
670.898665ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:13:51 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 670.921498ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration: 
408.776338ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:13:50 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 408.794914ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXX.X..........]
duration: 
640.52765ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:13:35 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 640.546985ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [........................XXXXXXXXXXXXXXXXXXXXXXXXX...........]
duration: 
502.541447ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:13:21 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 502.558538ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...................................................X........]
duration: 
540.735394ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:12:59 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 540.75432ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXX.X.........]
duration: 
795.634496ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:12:48 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 795.656758ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [.........................XXXXXXXXXXXXXXXXXXXXXXXXX..........]
duration: 
659.473431ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:12:35 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 659.488879ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXX.X........]
duration: 
782.001428ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:12:15 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 782.024072ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration: 
466.161107ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:11:57 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 466.190862ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [..........................XXXXXXXXXXXXXXXXXXXXXXXXX.........]
duration: 
560.157395ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:11:49 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 560.176961ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXX.X.......]
duration: 
646.712053ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:11:38 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 646.73241ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [...........................XXXXXXXXXXXXXXXXXXXXXXXXX........]
duration: 
659.948428ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:11:30 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 659.961022ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration: 
553.505909ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:11:12 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 553.528622ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXX.X......]
duration: 
785.970854ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:10:50 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 785.983858ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXX.X.....]
duration: 
702.038926ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:10:23 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 702.070064ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [............................XXXXXXXXXXXXXXXXXXXXXXXXX.......]
duration: 
764.272631ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:10:22 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 764.290925ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration: 
457.407252ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:10:20 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 457.455282ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXX.X....]
duration: 
668.427116ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:09:55 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 668.445842ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [.............................XXXXXXXXXXXXXXXXXXXXXXXXX......]
duration: 
542.102869ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:09:16 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 542.127164ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXX.X...]
duration: 
863.347567ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:09:16 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 863.37017ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration: 
449.389865ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:09:09 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 449.408921ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [........................................................X...]
duration: 
453.780278ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:08:38 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 453.801258ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [..............................XXXXXXXXXXXXXXXXXXXXXXXXX.....]
duration: 
829.20068ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:08:31 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 829.222619ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXX.X..]
duration: 
664.085955ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:08:07 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 664.134777ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [...............................XXXXXXXXXXXXXXXXXXXXXXXXX....]
duration: 
811.856639ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:08:03 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 811.870005ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration: 
537.134597ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:07:38 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 537.155635ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXX.X.]
duration: 
983.48601ms
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:07:35 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 983.504255ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [................................XXXXXXXXXXXXXXXXXXXXXXXXX...]
duration: 
931.23361ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:07:33 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 931.249529ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration: 
348.777729ms
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:07:04 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 348.796794ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [.................................XXXXXXXXXXXXXXXXXXXXXXXXX..]
duration: 
1.171586466s
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:06:40 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 1.171628436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
21 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXX.X]
duration: 
3.429689498s
trace: 
https://jaeger.dev.storj.io/trace/47dedb48446d33c1
FAIL (2025-12-15 12:06:39 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 3.429708754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................................X]
duration: 
30.018615481s
trace: 
https://jaeger.dev.storj.io/trace/335d64cce74e3214
FAIL (2025-12-15 12:06:26 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload
Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload
Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments...
Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false
Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment...
Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment.
Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece...
Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590     , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877     , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841     , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment
Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments...
Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618   , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438   , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337   , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447   , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518     , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802   , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907   , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015   , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977   , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663     , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367    , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555     , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil>
Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762   , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments...
Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.018640979s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
20 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXX.]
duration: 
5.699050907s
trace: 
https://jaeger.dev.storj.io/trace/29b9e72573ba3e3d
FAIL (2025-12-15 12:06:15 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816207B5FA15AE, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 5.699075432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [..................................XXXXXXXXXXXXXXXXXXXXXXXXX.]
duration: 
617.175927ms
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:05:45 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 617.206596ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX]
duration: 
4.822528856s
trace: 
https://jaeger.dev.storj.io/trace/29b9e72573ba3e3d
FAIL (2025-12-15 12:05:14 UTC) 
operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816207B5FA15AE, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10))
Duration: 4.822547913s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
25 out of 60 is failed (threshold=6) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX]
duration: 
5.000521781s
trace: 
https://jaeger.dev.storj.io/trace/140cd29facd4e598
FAIL (2025-12-15 12:04:56 UTC) 
operation error S3: PutObject, request canceled, context deadline exceeded
Duration: 5.000534174s core uplink us-select-1 us1-placement-20