Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [............................XXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 764.272631ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:10:22 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 764.290925ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 457.407252ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:10:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.455282ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXX.X....] duration: 668.427116ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:09:55 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 668.445842ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.............................XXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 542.102869ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:09:16 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 542.127164ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXX.X...] duration: 863.347567ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:09:16 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 863.37017ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 449.389865ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:09:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.408921ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 453.780278ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:08:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.801258ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..............................XXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 829.20068ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:08:31 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 829.222619ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXX.X..] duration: 664.085955ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:08:07 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 664.134777ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...............................XXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 811.856639ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:08:03 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 811.870005ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 537.134597ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:07:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 537.155635ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXX.X.] duration: 983.48601ms trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:07:35 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 983.504255ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [................................XXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 931.23361ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:07:33 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 931.249529ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 348.777729ms trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:07:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 348.796794ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.................................XXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 1.171586466s trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:06:40 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 1.171628436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXX.X] duration: 3.429689498s trace: https://jaeger.dev.storj.io/trace/47dedb48446d33c1 FAIL (2025-12-15 12:06:39 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881621B7F361BA2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.429708754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.018615481s trace: https://jaeger.dev.storj.io/trace/335d64cce74e3214 FAIL (2025-12-15 12:06:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 15 12:05:56.749470185 (236h36m44.447038532s) [upload=207961]: Starting upload Dec 15 12:05:56.749544054 (236h36m44.447112389s) [upload=207961]: Done starting upload Dec 15 12:05:56.749767430 (236h36m44.447335766s) [upload=207961]: Uploading segments... Dec 15 12:05:56.749958127 (236h36m44.447526463s) [upload=207961]: Got next segment. Inline: false Dec 15 12:05:56.750013350 (236h36m44.447581687s) [upload=207961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:05:56.750019502 (236h36m44.447587848s) [upload=207961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:05:56.797666136 (236h36m44.495234473s) [seg_pos={0 0}, upload=207961]: Begin upload segment... Dec 15 12:05:56.797704828 (236h36m44.495273164s) [seg_pos={0 0}, upload=207961]: Done begin upload segment. Dec 15 12:05:56.797745896 (236h36m44.495314241s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.797796150 (236h36m44.495364486s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798447166 (236h36m44.496015501s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.798995690 (236h36m44.496564026s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799074017 (236h36m44.496642362s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799436463 (236h36m44.49700481s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799746313 (236h36m44.497314648s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.799842131 (236h36m44.497410467s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800323250 (236h36m44.497891596s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800431692 (236h36m44.498000029s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.800806453 (236h36m44.498374799s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801016085 (236h36m44.49858442s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801225366 (236h36m44.498793701s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.801636644 (236h36m44.49920498s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.802364023 (236h36m44.499932359s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.803096520 (236h36m44.500664866s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Uploading piece... Dec 15 12:05:56.854745603 (236h36m44.552313949s) [noise=true , address=77.90.31.12:4590 , piece_id=4U36PUF54TCKASBIBIHXYKDGPAZYAGUN2CVS2VUAGYYFXYVXQIIQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856474272 (236h36m44.554042609s) [noise=true , address=77.90.31.21:4877 , piece_id=G73WQWYWMCEPCKE5654QCHGZ25TCUGTPG36PM4DCS2NGRD7GNDWQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.856729499 (236h36m44.554297836s) [noise=true , address=77.90.31.19:4841 , piece_id=7537WGH3U5LJRDCTFGBI4NIKVRD4VEPQGSFGYTGXU2U3G6QH4KEQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 12:05:56.914070765 (236h36m44.611639101s) [upload=207961]: Next returned nil segment Dec 15 12:05:56.914084190 (236h36m44.611652526s) [upload=207961]: Waiting for error group managing segments... Dec 15 12:05:56.928069265 (236h36m44.625637601s) [noise=true , address=145.79.227.25:4618 , piece_id=UAB6NPZOX4BKICPAOO2XOE5QASMI5CC54XNGJXT73J6S3QBGZT5Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929331853 (236h36m44.626900189s) [noise=true , address=145.79.227.20:4438 , piece_id=4L6DGNWJR3Z57EW3OMNTNXHFWSNSMOVTX2YVFMO3WTAHKT6RCOQA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.929375425 (236h36m44.626943761s) [noise=true , address=145.79.227.49:4337 , piece_id=KOW5EWZU6U5P77WKUHAKDA557YCNVO2OMZBPC3LE4GTDY5BDHAMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930011252 (236h36m44.627579588s) [noise=true , address=145.79.227.52:4447 , piece_id=HG7MJFAZEKTYP57A52KHYS55G5U4AWKHC2JJBS4PXCPSRXORW57Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930187632 (236h36m44.627755978s) [noise=true , address=77.90.31.10:4518 , piece_id=FTVDWDRNK6GBJ2TDYCED4JOHMP64SZQNS4SHBYGPHBLYVFLNDCMQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930582028 (236h36m44.628150365s) [noise=true , address=145.79.227.13:4802 , piece_id=YAWZYSRM44KOTQO74MJNBFMB3LOODWIQICF46KVQTBV3JIX7OQVQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.930801609 (236h36m44.628369945s) [noise=true , address=145.79.227.34:4907 , piece_id=JUIWXGXIAVQF4QWPBUGG47GI3WYXNUE6DMTOJGJTWP3EO2RSUMZA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931344463 (236h36m44.628912799s) [noise=true , address=145.79.227.39:4015 , piece_id=I22GWL3PB2TIRPOMK72LP2DKVIPCZ2MFLNN4MD3EHMCZ6X4AQ7QA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931489714 (236h36m44.629058051s) [noise=true , address=145.79.227.38:4977 , piece_id=GIGVPLJDSMQT6UMPLEOMHFUNXI5DHSSZJVU3P6QPHU6GXZ5633LQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.931962767 (236h36m44.629531104s) [noise=true , address=77.90.31.14:4663 , piece_id=42DU462B3UGWGR3QBBSSCAESUEL5SODQKCL52YYSVPCUGNLVW56A, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:56.932184141 (236h36m44.629752478s) [noise=true , address=145.79.227.9:4367 , piece_id=X3ADNSXPCUVPAKKD6IZQGPYYZQY5VSTHKWQQ6YHRGZ67HZ3U7BTA, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:05:57.129527226 (236h36m44.827095563s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOH4OXBNJPUIJSC3S4X2DWK73AXF567RTSMJFFDOTMUEBSEYJAQ, seg_pos={0 0}, upload=207961]: Done uploading piece. err: <nil> Dec 15 12:06:26.704458766 (236h37m14.402027102s) [noise=true , address=145.79.227.30:4762 , piece_id=TEBIX4WUG2SWR5KKA7Z6MMYPHIJPSHBAKQKT3XMAZQBS7YLW632Q, seg_pos={0 0}, upload=207961]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded Dec 15 12:06:26.704503140 (236h37m14.402071475s) [seg_pos={0 0}, upload=207961]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 15 12:06:26.719801077 (236h37m14.417369413s) [upload=207961]: Done uploading segments... Dec 15 12:06:26.719830712 (236h37m14.417399049s) [upload=207961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.018640979s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXX.] duration: 5.699050907s trace: https://jaeger.dev.storj.io/trace/29b9e72573ba3e3d FAIL (2025-12-15 12:06:15 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816207B5FA15AE, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 5.699075432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..................................XXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 617.175927ms trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:05:45 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 617.206596ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 4.822528856s trace: https://jaeger.dev.storj.io/trace/29b9e72573ba3e3d FAIL (2025-12-15 12:05:14 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816207B5FA15AE, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 4.822547913s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000521781s trace: https://jaeger.dev.storj.io/trace/140cd29facd4e598 FAIL (2025-12-15 12:04:56 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 5.000534174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1.839329765s trace: https://jaeger.dev.storj.io/trace/2caf4ea2a305855e FAIL (2025-12-15 12:04:07 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161F81F144A74, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 1.839345985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 4.003186817s trace: https://jaeger.dev.storj.io/trace/29f2dd2370f30917 FAIL (2025-12-15 12:03:47 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161F3999FA551, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 4.003202045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 2.95675903s trace: https://jaeger.dev.storj.io/trace/732b6cfdaa0f73ce FAIL (2025-12-15 12:03:21 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161ED5F37D456, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 2.956782073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 2.159611267s trace: https://jaeger.dev.storj.io/trace/636796ba2ef97454 FAIL (2025-12-15 12:03:20 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161ED5152D8D5, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 2.159630673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 4.802006138s trace: https://jaeger.dev.storj.io/trace/3604d064eb1fdb9a FAIL (2025-12-15 12:02:49 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161E5FC9B53CD, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 4.802022849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 4.389898741s trace: https://jaeger.dev.storj.io/trace/4077130f90cfc5c6 FAIL (2025-12-15 12:02:25 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161E05C7B48E2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 4.389916905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 3.82015209s trace: https://jaeger.dev.storj.io/trace/68d9ae0d7f0a3443 FAIL (2025-12-15 12:01:56 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161D9C4B736B2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.820169891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 2.625475563s trace: https://jaeger.dev.storj.io/trace/550ed53e0037ebff FAIL (2025-12-15 12:01:40 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161D5EE6A8E6E, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 2.625490261s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 3.70614338s trace: https://jaeger.dev.storj.io/trace/7d6bdd4f29e01159 FAIL (2025-12-15 12:01:34 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161D48EF8ADE9, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.706161805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 2.638764415s trace: https://jaeger.dev.storj.io/trace/ced699de8c9485 FAIL (2025-12-15 12:00:52 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161CAC9869C97, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 2.638780796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 5.000426184s trace: https://jaeger.dev.storj.io/trace/5f949a2555e7d625 FAIL (2025-12-15 12:00:28 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 5.000442845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 2.374433172s trace: https://jaeger.dev.storj.io/trace/373a5b8a8e88d55c FAIL (2025-12-15 12:00:22 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161C3B98CDDAC, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 2.374450644s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 3.13354771s trace: https://jaeger.dev.storj.io/trace/4d61158984b70883 FAIL (2025-12-15 12:00:03 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161BF702BFEF8, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.133570282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..........................................XXXXXXXXXXXXXXXXXX] duration: 3.019959615s trace: https://jaeger.dev.storj.io/trace/224d2ee5ed8eb005 FAIL (2025-12-15 11:59:33 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161B85BA131C8, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.019977909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...........................................XXXXXXXXXXXXXXXXX] duration: 3.29630218s trace: https://jaeger.dev.storj.io/trace/5179beadf6ed38e7 FAIL (2025-12-15 11:59:07 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161B2643524F3, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.296313632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 2.959041196s trace: https://jaeger.dev.storj.io/trace/5da42432f3a82ba2 FAIL (2025-12-15 11:59:00 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161B09DA4094B, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 2.959060973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [............................................XXXXXXXXXXXXXXXX] duration: 4.355120675s trace: https://jaeger.dev.storj.io/trace/248dd45b8013561e FAIL (2025-12-15 11:58:32 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161AA434E669E, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 4.355138288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.............................................XXXXXXXXXXXXXXX] duration: 1.77629941s trace: https://jaeger.dev.storj.io/trace/5f4c312cef0081ae FAIL (2025-12-15 11:58:00 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161A2B216F8BD, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 1.776316832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 4.607820627s trace: https://jaeger.dev.storj.io/trace/3512805241d79516 FAIL (2025-12-15 11:57:50 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161A05D2E2CBF, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 4.607842367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..............................................XXXXXXXXXXXXXX] duration: 4.89366436s trace: https://jaeger.dev.storj.io/trace/2bfc754cdc4b5f07 FAIL (2025-12-15 11:57:34 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881619C8BA74F0C, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 11)) Duration: 4.893687704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...............................................XXXXXXXXXXXXX] duration: 1.04318517s trace: https://jaeger.dev.storj.io/trace/545e2278a9c2c154 FAIL (2025-12-15 11:57:02 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881619537E6A7F5, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 1.043208144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 3.584848052s trace: https://jaeger.dev.storj.io/trace/5b3305ef1d6b362 FAIL (2025-12-15 11:56:51 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881619296E725D3, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.58490536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [................................................XXXXXXXXXXXX] duration: 3.590706917s trace: https://jaeger.dev.storj.io/trace/771be5f8014a1711 FAIL (2025-12-15 11:56:32 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881618E4EA7C78D, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 11)) Duration: 3.590725462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 3.811810815s trace: https://jaeger.dev.storj.io/trace/4801ea9a872881f8 FAIL (2025-12-15 11:55:50 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161845C99966B, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.811843606s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.................................................XXXXXXXXXXX] duration: 2.913714223s trace: https://jaeger.dev.storj.io/trace/7806d48739005e60 FAIL (2025-12-15 11:55:46 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881618397B3AB5B, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 2.913740292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..................................................XXXXXXXXXX] duration: 5.000399777s trace: https://jaeger.dev.storj.io/trace/7d8da6d588930dc6 FAIL (2025-12-15 11:55:21 UTC) operation error S3: PutObject, request canceled, context deadline exceeded Duration: 5.000412971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...................................................XXXXXXXXX] duration: 2.048662175s trace: https://jaeger.dev.storj.io/trace/78ce2af96503f332 FAIL (2025-12-15 11:54:50 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816176761F4EC3, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 2.048681813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 1.769096163s trace: https://jaeger.dev.storj.io/trace/6ffb98e593d0f15d FAIL (2025-12-15 11:54:46 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816175817CA63A, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 1.76911537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....................................................XXXXXXXX] duration: 1.53831085s trace: https://jaeger.dev.storj.io/trace/2f93d69f34e7862 FAIL (2025-12-15 11:54:01 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881616B02202B0E, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 1.538328152s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)