Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.018169128s trace: https://jaeger.dev.storj.io/trace/31217081d93457fc FAIL (2025-12-10 23:37:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Item_ObjectBegin Dec 10 23:36:35.656095130 (128h7m23.353663446s) [upload=112474]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:36:35.679405072 (128h7m23.376973378s) [seg_pos={0 0}, upload=112474]: Begin upload segment... Dec 10 23:36:35.679433145 (128h7m23.377001461s) [seg_pos={0 0}, upload=112474]: Done begin upload segment. Dec 10 23:36:35.679451740 (128h7m23.377020056s) [noise=true , address=45.91.116.9:4274 , piece_id=NSNLVVB2GJEN7ADLGSMGURRILLGPFNZHAZ4ZC4DV32DGX4ERN4CA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.679544753 (128h7m23.377113059s) [noise=true , address=77.90.31.10:4543 , piece_id=ZRM7I4WJT2EJMUDKJY7DORN7N6QFJI7SOAO5HQLN34LRUOO7GWAA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.679995004 (128h7m23.37756331s) [noise=true , address=77.90.31.11:4576 , piece_id=WKLQAFAGOSD7YMCD2RJQTPKK7U4OAT5VPS5OXAEBRQJB7G2J7E2A, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680110460 (128h7m23.377678776s) [noise=true , address=145.79.227.44:4206 , piece_id=2Q3RN4O3VKMVQ6VTXOUPWLXUKCE5ER4D5EEETL6A3VU2SRDFQHNA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680288923 (128h7m23.377857239s) [noise=true , address=145.79.227.25:4639 , piece_id=EY2DYVHDDCYUV4XLJUNDSMZAGVHVQMLT44M7GVKLGPMMBIDD5P2A, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680517830 (128h7m23.378086137s) [noise=true , address=45.91.116.12:4256 , piece_id=6PNLP44652EJHVGEPO3EHXR6G6YDGDHFIT7KFY4D2SRFPP7ONVTA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680576681 (128h7m23.378144997s) [noise=true , address=5.231.73.5:4020 , piece_id=44WLLWHL47EUSOYX7F4QKSBPBIVPL4KUFJQXFBFT3STUSZ3PMQEA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680843670 (128h7m23.378411986s) [noise=true , address=77.90.31.12:4616 , piece_id=CSZIHDDAFW7U37QY5VFOE2VGFRDOMFUJ6TG7NUUKC6WTMWIYKSQA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680902820 (128h7m23.378471137s) [noise=true , address=45.91.116.14:4223 , piece_id=ANIFCTKTPO3LFMMGDCUJVVSGYTNBHOYIMRNSG3A7X35T6TTLP2VA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681088006 (128h7m23.378656312s) [noise=true , address=145.79.227.30:4788 , piece_id=5UY6COIJARBN3IGEZ23Z6Q4EJXRHMCPVIQEI2UQKZAHKK2PIPT3Q, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681197941 (128h7m23.378766247s) [noise=true , address=77.90.31.13:4655 , piece_id=F5DXTRR6ZEE6WII7PXHU4DBXJR5Y26G3BIF57I4YF2S3QOWO3VLA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681413634 (128h7m23.37898199s) [noise=true , address=145.79.227.34:4931 , piece_id=JZCPQGSSAR5VGY23TRH3TNFTQCPXFKXQMZDQSC6BQFF3F65TTN4A, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681557122 (128h7m23.379125438s) [noise=true , address=77.90.31.16:4746 , piece_id=RYLC4LB75FHWGGJ5H2BLEND7DJ2U2Z3FGVPUBJT5UMDYVE5JPWCQ, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681751856 (128h7m23.379320172s) [noise=true , address=145.79.227.12:4775 , piece_id=VULPNF6Q6IM354KY4QAMCDAMQS6INXGP73IHP2GCEHUV7KZFZYNA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681848427 (128h7m23.379416743s) [noise=true , address=45.91.116.5:4300 , piece_id=WOK3AMMF3KWIM5FKNEYVSO5ZIZZ2MSCJIGYL65RIOZXBZ4LG2VZA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681998206 (128h7m23.379566522s) [noise=true , address=77.90.31.19:4872 , piece_id=N5KG52SCRGEQ5QMZ6732NQAZJL2YQJBDFEFWKF5NKJSNZC36SAFA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.682153988 (128h7m23.379722293s) [noise=true , address=145.79.227.31:4815 , piece_id=B3T6RYWLCQEE5DWZVZMLW4C65U3WKBH55VD2B7JP4V7XO3GZJMZQ, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.682562711 (128h7m23.380131017s) [noise=true , address=145.79.227.44:4222 , piece_id=OTRFBYPUBHR3TDSMYPE3GGIC7M3A6A4VLW23MHPPN53B5GXOU2GA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.756175738 (128h7m23.453744054s) [upload=112474]: Next returned nil segment Dec 10 23:36:35.756310098 (128h7m23.453878405s) [upload=112474]: Waiting for error group managing segments... Dec 10 23:36:35.766788391 (128h7m23.464356707s) [noise=true , address=45.91.116.5:4300 , piece_id=WOK3AMMF3KWIM5FKNEYVSO5ZIZZ2MSCJIGYL65RIOZXBZ4LG2VZA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.767580251 (128h7m23.465148567s) [noise=true , address=45.91.116.9:4274 , piece_id=NSNLVVB2GJEN7ADLGSMGURRILLGPFNZHAZ4ZC4DV32DGX4ERN4CA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.767762311 (128h7m23.465330627s) [noise=true , address=45.91.116.12:4256 , piece_id=6PNLP44652EJHVGEPO3EHXR6G6YDGDHFIT7KFY4D2SRFPP7ONVTA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.770602847 (128h7m23.468171163s) [noise=true , address=145.79.227.25:4639 , piece_id=EY2DYVHDDCYUV4XLJUNDSMZAGVHVQMLT44M7GVKLGPMMBIDD5P2A, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.776445361 (128h7m23.474013677s) [noise=true , address=145.79.227.31:4815 , piece_id=B3T6RYWLCQEE5DWZVZMLW4C65U3WKBH55VD2B7JP4V7XO3GZJMZQ, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.794245652 (128h7m23.491813957s) [noise=true , address=145.79.227.34:4931 , piece_id=JZCPQGSSAR5VGY23TRH3TNFTQCPXFKXQMZDQSC6BQFF3F65TTN4A, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.794912827 (128h7m23.492481143s) [noise=true , address=145.79.227.12:4775 , piece_id=VULPNF6Q6IM354KY4QAMCDAMQS6INXGP73IHP2GCEHUV7KZFZYNA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.801998473 (128h7m23.499566789s) [noise=true , address=45.91.116.14:4223 , piece_id=ANIFCTKTPO3LFMMGDCUJVVSGYTNBHOYIMRNSG3A7X35T6TTLP2VA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.863391334 (128h7m23.56095965s) [noise=true , address=5.231.73.5:4020 , piece_id=44WLLWHL47EUSOYX7F4QKSBPBIVPL4KUFJQXFBFT3STUSZ3PMQEA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:36.695861673 (128h7m24.393430009s) [noise=true , address=145.79.227.44:4222 , piece_id=OTRFBYPUBHR3TDSMYPE3GGIC7M3A6A4VLW23MHPPN53B5GXOU2GA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:36.723990597 (128h7m24.421558933s) [noise=true , address=145.79.227.44:4206 , piece_id=2Q3RN4O3VKMVQ6VTXOUPWLXUKCE5ER4D5EEETL6A3VU2SRDFQHNA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:36.727710947 (128h7m24.425279283s) [noise=true , address=145.79.227.30:4788 , piece_id=5UY6COIJARBN3IGEZ23Z6Q4EJXRHMCPVIQEI2UQKZAHKK2PIPT3Q, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:37:35.639633810 (128h8m23.337202146s) [noise=true , address=77.90.31.13:4655 , piece_id=F5DXTRR6ZEE6WII7PXHU4DBXJR5Y26G3BIF57I4YF2S3QOWO3VLA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD, address:77.90.31.13:4655): protocol: rpc: dial tcp 77.90.31.13:4655: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.639787307 (128h8m23.337355642s) [noise=true , address=77.90.31.16:4746 , piece_id=RYLC4LB75FHWGGJ5H2BLEND7DJ2U2Z3FGVPUBJT5UMDYVE5JPWCQ, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: rpc: dial tcp 77.90.31.16:4746: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.639892653 (128h8m23.337460989s) [noise=true , address=77.90.31.19:4872 , piece_id=N5KG52SCRGEQ5QMZ6732NQAZJL2YQJBDFEFWKF5NKJSNZC36SAFA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL, address:77.90.31.19:4872): protocol: rpc: dial tcp 77.90.31.19:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.640142139 (128h8m23.337710475s) [noise=true , address=77.90.31.10:4543 , piece_id=ZRM7I4WJT2EJMUDKJY7DORN7N6QFJI7SOAO5HQLN34LRUOO7GWAA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 5.161.89.252:41702->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.640177455 (128h8m23.337745801s) [noise=true , address=77.90.31.11:4576 , piece_id=WKLQAFAGOSD7YMCD2RJQTPKK7U4OAT5VPS5OXAEBRQJB7G2J7E2A, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: rpc: dial tcp 77.90.31.11:4576: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.640463459 (128h8m23.338031796s) [noise=true , address=77.90.31.12:4616 , piece_id=CSZIHDDAFW7U37QY5VFOE2VGFRDOMFUJ6TG7NUUKC6WTMWIYKSQA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, address:77.90.31.12:4616): protocol: write tcp 5.161.89.252:39650->77.90.31.12:4616: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.640559238 (128h8m23.338127584s) [seg_pos={0 0}, upload=112474]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:37:35.655262553 (128h8m23.35283089s) [upload=112474]: Done uploading segments... Dec 10 23:37:35.655286949 (128h8m23.352855285s) [upload=112474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.018180159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XXX................] duration: 626.47754ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:37:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 626.491155ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................................XXX...............] duration: 605.311624ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:37:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.332684ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X..X.X....X.X.....] duration: 413.286183ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:37:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 413.403083ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................................XX.XX................] duration: 475.905788ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:36:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 475.930224ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.014730516s trace: https://jaeger.dev.storj.io/trace/3eeb869daa6c330e FAIL (2025-12-10 23:36:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== .90.31.17:4775 , piece_id=Y6GXBPJ2DNMHTDAQXRNVJ2SQEKRXUMTA2XDGTKNT7BS3XI74RIYQ, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.624665781 (128h6m23.322234117s) [noise=true , address=77.90.31.16:4746 , piece_id=2CPZYMFCIDRZ2L3NFNIZUK3ILM245YQNGBC5X3YDZ7DPWH3EGAIA, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.625363083 (128h6m23.32293143s) [noise=true , address=77.90.31.11:4558 , piece_id=WFK3SLR5DKTDROGNXABJQYKQEQH7IZKJRH73GSCN4UBEWMIWGHIQ, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.626067420 (128h6m23.323635765s) [noise=true , address=145.79.227.40:4059 , piece_id=5IWFCFURBGKTGRS7QINXH3EAWY23LACO2XUAMSKOENM73MM36DMQ, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.626758851 (128h6m23.324327187s) [noise=true , address=145.79.227.52:4466 , piece_id=23DU326YQL6YCFKJ6WK6SLCSTWJTEFWRZSOU6K35LWFPD6QJOYZQ, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.627221354 (128h6m23.32478969s) [noise=true , address=77.90.31.21:4910 , piece_id=XBQCRFA6VXGVW7C3JDK2PFKVPAOLEPN7CS32Y7GIX52GZ5MPZEKA, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.627694358 (128h6m23.325262694s) [noise=true , address=45.91.116.4:4962 , piece_id=726RWBXVDID4JAYE7DIBTES7DDLE6M5UEUZK5MY7GHTORNJJRAPQ, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.628224678 (128h6m23.325793015s) [noise=true , address=77.90.31.15:4709 , piece_id=NT5YGJ4YBK5JQY5TKM3N2IG2B2OQWHRLZDBYCQEXKLTL3ANOX3XA, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.630434536 (128h6m23.328002872s) [noise=true , address=5.231.73.4:4035 , piece_id=666PVK3URLI5VCY7UF2UD6VEU2E665F6RPKTIX7V6DJRWQEXRSJA, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.630977081 (128h6m23.328545416s) [noise=true , address=45.91.116.14:4220 , piece_id=FY42XRAAWU5RA5GX7N4PA75CIV2PBZKQ5Q3NV3T244KCDLASHW3Q, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.631531446 (128h6m23.329099782s) [noise=true , address=77.90.31.18:4830 , piece_id=CQYD7KAMMTNIXUOFC42P3BH2EXPFRFLCJGYUIKENMR2IDYUCZ7TA, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.633323844 (128h6m23.33089218s) [noise=true , address=145.79.227.16:4929 , piece_id=SK4R3XCYP2QGNHAL3FC4WTB6SC4TKY4RGZMDO57PD6MDADLFOOXA, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.635116312 (128h6m23.332684648s) [noise=true , address=77.90.31.13:4625 , piece_id=36VCPSI63ZTUEYHMSUGOM5WC436ZUHQKYAGQ6ZQA32Z3BU3S3L3A, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.637434361 (128h6m23.335002708s) [noise=true , address=145.79.227.7:4326 , piece_id=X7QUUBIMMDYDTSDVYG66GESNOMWXBPVLFNZ32H5JLPZ6B7EEDHOQ, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.641039056 (128h6m23.338607392s) [noise=true , address=77.90.31.19:4876 , piece_id=N7JA7L5JX4ICUGCE5BSLFIWRIF6U6ETVPZFYYXI5S76VSS7OABTQ, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.642929477 (128h6m23.340497813s) [noise=true , address=77.90.31.10:4533 , piece_id=WSP2UKFPXBEARGSYO7XBPXV6CGNXWKJOD3G4ZSOLXCLYZE7H2GQA, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.644752532 (128h6m23.342320868s) [noise=true , address=5.231.73.3:4015 , piece_id=OJRBDYUJCROH3J6US36D472JBAXH3QMXXSRDV3CPQ7YUQJXSJKAQ, seg_pos={0 0}, upload=112459]: Uploading piece... Dec 10 23:35:35.726237442 (128h6m23.423805829s) [upload=112459]: Next returned nil segment Dec 10 23:35:35.726258873 (128h6m23.423827209s) [upload=112459]: Waiting for error group managing segments... Dec 10 23:35:35.738088259 (128h6m23.435656595s) [noise=true , address=45.91.116.4:4962 , piece_id=726RWBXVDID4JAYE7DIBTES7DDLE6M5UEUZK5MY7GHTORNJJRAPQ, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:35:35.740518840 (128h6m23.438087186s) [noise=true , address=145.79.227.52:4466 , piece_id=23DU326YQL6YCFKJ6WK6SLCSTWJTEFWRZSOU6K35LWFPD6QJOYZQ, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:35:35.749914932 (128h6m23.447483248s) [noise=true , address=145.79.227.7:4326 , piece_id=X7QUUBIMMDYDTSDVYG66GESNOMWXBPVLFNZ32H5JLPZ6B7EEDHOQ, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:35:35.752314404 (128h6m23.44988274s) [noise=true , address=145.79.227.2:4146 , piece_id=GKBTX7C6EWXKDDVJF7FTYUCUYE4Q35ZGCJFZIJ574C2UMZ2ABVRA, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:35:35.755292476 (128h6m23.452860812s) [noise=true , address=45.91.116.14:4220 , piece_id=FY42XRAAWU5RA5GX7N4PA75CIV2PBZKQ5Q3NV3T244KCDLASHW3Q, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:35:35.796600619 (128h6m23.494168955s) [noise=true , address=145.79.227.40:4059 , piece_id=5IWFCFURBGKTGRS7QINXH3EAWY23LACO2XUAMSKOENM73MM36DMQ, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:35:35.873371584 (128h6m23.57093992s) [noise=true , address=5.231.73.4:4035 , piece_id=666PVK3URLI5VCY7UF2UD6VEU2E665F6RPKTIX7V6DJRWQEXRSJA, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:35:35.887064291 (128h6m23.584632628s) [noise=true , address=5.231.73.3:4015 , piece_id=OJRBDYUJCROH3J6US36D472JBAXH3QMXXSRDV3CPQ7YUQJXSJKAQ, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:35:36.784006164 (128h6m24.48157447s) [noise=true , address=145.79.227.16:4929 , piece_id=SK4R3XCYP2QGNHAL3FC4WTB6SC4TKY4RGZMDO57PD6MDADLFOOXA, seg_pos={0 0}, upload=112459]: Done uploading piece. err: <nil> Dec 10 23:36:35.576097406 (128h7m23.273665722s) [noise=true , address=77.90.31.18:4830 , piece_id=CQYD7KAMMTNIXUOFC42P3BH2EXPFRFLCJGYUIKENMR2IDYUCZ7TA, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: rpc: dial tcp 77.90.31.18:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576111423 (128h7m23.273679728s) [noise=true , address=77.90.31.11:4558 , piece_id=WFK3SLR5DKTDROGNXABJQYKQEQH7IZKJRH73GSCN4UBEWMIWGHIQ, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: rpc: dial tcp 77.90.31.11:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576221257 (128h7m23.273789563s) [noise=true , address=77.90.31.17:4775 , piece_id=Y6GXBPJ2DNMHTDAQXRNVJ2SQEKRXUMTA2XDGTKNT7BS3XI74RIYQ, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:127VKvwi82kYfGZ6Wcv8BinaNUQBMQsgKLKNNLht5JPsBLN7ovt, address:77.90.31.17:4775): protocol: rpc: dial tcp 77.90.31.17:4775: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576266442 (128h7m23.273834758s) [noise=true , address=77.90.31.13:4625 , piece_id=36VCPSI63ZTUEYHMSUGOM5WC436ZUHQKYAGQ6ZQA32Z3BU3S3L3A, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: rpc: dial tcp 77.90.31.13:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576330913 (128h7m23.273899219s) [noise=true , address=77.90.31.16:4746 , piece_id=2CPZYMFCIDRZ2L3NFNIZUK3ILM245YQNGBC5X3YDZ7DPWH3EGAIA, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: rpc: dial tcp 77.90.31.16:4746: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576393520 (128h7m23.273961836s) [noise=true , address=77.90.31.10:4533 , piece_id=WSP2UKFPXBEARGSYO7XBPXV6CGNXWKJOD3G4ZSOLXCLYZE7H2GQA, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: rpc: dial tcp 77.90.31.10:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576434696 (128h7m23.274003003s) [noise=true , address=77.90.31.15:4709 , piece_id=NT5YGJ4YBK5JQY5TKM3N2IG2B2OQWHRLZDBYCQEXKLTL3ANOX3XA, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: rpc: dial tcp 77.90.31.15:4709: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576579297 (128h7m23.274147603s) [noise=true , address=77.90.31.21:4910 , piece_id=XBQCRFA6VXGVW7C3JDK2PFKVPAOLEPN7CS32Y7GIX52GZ5MPZEKA, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: rpc: dial tcp 77.90.31.21:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576708819 (128h7m23.274277135s) [noise=true , address=77.90.31.19:4876 , piece_id=N7JA7L5JX4ICUGCE5BSLFIWRIF6U6ETVPZFYYXI5S76VSS7OABTQ, seg_pos={0 0}, upload=112459]: Done uploading piece. err: ecclient: upload failed (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB, address:77.90.31.19:4876): protocol: write tcp 5.161.89.252:57634->77.90.31.19:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:36:35.576728125 (128h7m23.274296431s) [seg_pos={0 0}, upload=112459]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Dec 10 23:36:35.589868731 (128h7m23.287437047s) [upload=112459]: Done uploading segments... Dec 10 23:36:35.589883639 (128h7m23.287451955s) [upload=112459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.014746486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................................XX.XX...............] duration: 482.217828ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:36:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.234979ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................................XXX..............] duration: 808.92067ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:36:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 808.944795ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................X..X.X....X.X....] duration: 444.853643ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:35:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.877307ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.022058977s trace: https://jaeger.dev.storj.io/trace/1fb9e772d7d65c1e FAIL (2025-12-10 23:35:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 93 (128h5m23.213395828s) [upload=112449]: Uploading segments... Dec 10 23:34:35.516185582 (128h5m23.213753918s) [upload=112449]: Got next segment. Inline: false Dec 10 23:34:35.516248058 (128h5m23.213816394s) [upload=112449]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:34:35.516254490 (128h5m23.213822826s) [upload=112449]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:34:35.540443075 (128h5m23.23801141s) [seg_pos={0 0}, upload=112449]: Begin upload segment... Dec 10 23:34:35.540504489 (128h5m23.238072826s) [seg_pos={0 0}, upload=112449]: Done begin upload segment. Dec 10 23:34:35.540536037 (128h5m23.238104384s) [noise=true , address=77.90.31.13:4637 , piece_id=FXR6KRLUJWJNZXJSRI72YNUE24SDWE5AT3T4WJMBMN5M2GUCJPEQ, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.540598385 (128h5m23.238166721s) [noise=true , address=5.231.73.3:4027 , piece_id=3CB6GDOVBZWQ7J6XRTHINVEHIVI4VPIICBMRVN4NWCG4C3ATFTYA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.541140908 (128h5m23.238709254s) [noise=true , address=145.79.227.39:4041 , piece_id=ADH7JPPPXLLFEVOGUMFY3TNAVJIAGL4A4PTTUOHO5MUULGOWGJSQ, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.541655118 (128h5m23.239223464s) [noise=true , address=45.91.116.14:4213 , piece_id=OWTFTYJAPFT2556PWTTDM7W345BHSQMF765HCQEZHKSU7RIKP4GA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.542150544 (128h5m23.23971888s) [noise=true , address=145.79.227.7:4313 , piece_id=QBRE3T6LHL6TDICHLH4W5SU3JQZPLERA3NJPZM2VG3RPSIQEE4CQ, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.542164370 (128h5m23.239732706s) [noise=true , address=77.90.31.17:4782 , piece_id=A4CHRLRTQLWFPXQRQFHOSP4V2ORYHQPY3MGO6GFLWCEEMN2ODN6Q, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.542601397 (128h5m23.240169732s) [noise=true , address=45.91.116.12:4228 , piece_id=6YEA26WS62Z3UP34NHOIWNWZWJ33BTEXDEEKJ7HUGQ6JPQYU6CLA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.542629218 (128h5m23.240197554s) [noise=true , address=45.91.116.11:4146 , piece_id=LCRJFTFXGGSCADX4CA6APNZXGG7RBP5KQUHFOWSL3CXYJM6WIZ4A, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.543007195 (128h5m23.24057553s) [noise=true , address=64.49.13.169:4710 , piece_id=J5TOSXUGQYGLT2DCRYGWZGJR2O3LJXS7AIB6LXLCKEWNIKSLAKKA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.543047940 (128h5m23.240616276s) [noise=true , address=45.91.116.10:4094 , piece_id=PCKKWD6ZNL33LN5WXMU2XFEQR62N2AKF6V53POT3QWNO37ABFCGA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.543401942 (128h5m23.240970278s) [noise=true , address=45.91.116.8:4058 , piece_id=QDEY46HWXSPZYECNJWQT6CZKCHS5MLLGCF3CP6KFD76HRSY3SLEA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.543464868 (128h5m23.241033205s) [noise=true , address=45.91.116.13:4161 , piece_id=C34E7IO2FFQ7PBX7O4SKFP5EY55HR37YNYDXI2Q42JDN2NFN3HWQ, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.543851923 (128h5m23.241420258s) [noise=true , address=77.90.31.15:4703 , piece_id=INX7IHUCBJ7OZMS6EVPLBLWRTNHSMETAMFPX5PXJ4FDHVNSIZ5FA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.543949294 (128h5m23.24151763s) [noise=true , address=145.79.227.4:4193 , piece_id=4MHKLFTSKAO2YEDVFN2O72HD5YPBW4EEDWLXLPHSYINRSCWV3HZA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.544314215 (128h5m23.241882561s) [noise=true , address=77.90.31.16:4739 , piece_id=BMO5MBOSKYVKSELFE5NODVPQ5LJFQHQWN5N5JBDHSPCOAPS6C6KA, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.544760359 (128h5m23.242328695s) [noise=true , address=77.90.31.19:4847 , piece_id=DRCXXVCKY653OEMAFDMDRGEJ4C6ZH5H7MIESMUXZSHXRQL5SJNQQ, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.545173961 (128h5m23.242742297s) [noise=true , address=145.79.227.27:4720 , piece_id=SZNLQPGV5FEVB2Q3VNJAVAGSJLG2FUM3GGIY65HQ3XDOP2ICKWOQ, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.546400953 (128h5m23.24396928s) [noise=true , address=45.91.116.4:4950 , piece_id=J4J6WZQB45KH6EYIVCO7GG3HFXT7LOS2FZXGQIR2NCHVOR7OHKEQ, seg_pos={0 0}, upload=112449]: Uploading piece... Dec 10 23:34:35.617221945 (128h5m23.314790261s) [upload=112449]: Next returned nil segment Dec 10 23:34:35.617230420 (128h5m23.314798736s) [upload=112449]: Waiting for error group managing segments... Dec 10 23:34:35.625519383 (128h5m23.323087689s) [noise=true , address=45.91.116.10:4094 , piece_id=PCKKWD6ZNL33LN5WXMU2XFEQR62N2AKF6V53POT3QWNO37ABFCGA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.629156647 (128h5m23.326724954s) [noise=true , address=145.79.227.39:4041 , piece_id=ADH7JPPPXLLFEVOGUMFY3TNAVJIAGL4A4PTTUOHO5MUULGOWGJSQ, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.637652859 (128h5m23.335221175s) [noise=true , address=45.91.116.13:4161 , piece_id=C34E7IO2FFQ7PBX7O4SKFP5EY55HR37YNYDXI2Q42JDN2NFN3HWQ, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.638153734 (128h5m23.33572204s) [noise=true , address=45.91.116.4:4950 , piece_id=J4J6WZQB45KH6EYIVCO7GG3HFXT7LOS2FZXGQIR2NCHVOR7OHKEQ, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.641622984 (128h5m23.33919129s) [noise=true , address=45.91.116.12:4228 , piece_id=6YEA26WS62Z3UP34NHOIWNWZWJ33BTEXDEEKJ7HUGQ6JPQYU6CLA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.643501995 (128h5m23.341070301s) [noise=true , address=145.79.227.27:4720 , piece_id=SZNLQPGV5FEVB2Q3VNJAVAGSJLG2FUM3GGIY65HQ3XDOP2ICKWOQ, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.643817033 (128h5m23.341385349s) [noise=true , address=45.91.116.11:4146 , piece_id=LCRJFTFXGGSCADX4CA6APNZXGG7RBP5KQUHFOWSL3CXYJM6WIZ4A, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.646946580 (128h5m23.344514885s) [noise=true , address=45.91.116.8:4058 , piece_id=QDEY46HWXSPZYECNJWQT6CZKCHS5MLLGCF3CP6KFD76HRSY3SLEA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.657254845 (128h5m23.354823151s) [noise=true , address=45.91.116.14:4213 , piece_id=OWTFTYJAPFT2556PWTTDM7W345BHSQMF765HCQEZHKSU7RIKP4GA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.703689313 (128h5m23.401257619s) [noise=true , address=145.79.227.4:4193 , piece_id=4MHKLFTSKAO2YEDVFN2O72HD5YPBW4EEDWLXLPHSYINRSCWV3HZA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:35.728645300 (128h5m23.426213606s) [noise=true , address=5.231.73.3:4027 , piece_id=3CB6GDOVBZWQ7J6XRTHINVEHIVI4VPIICBMRVN4NWCG4C3ATFTYA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:36.236771779 (128h5m23.934340085s) [noise=true , address=145.79.227.7:4313 , piece_id=QBRE3T6LHL6TDICHLH4W5SU3JQZPLERA3NJPZM2VG3RPSIQEE4CQ, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:34:41.364931488 (128h5m29.062499804s) [noise=true , address=64.49.13.169:4710 , piece_id=J5TOSXUGQYGLT2DCRYGWZGJR2O3LJXS7AIB6LXLCKEWNIKSLAKKA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: <nil> Dec 10 23:35:35.502633109 (128h6m23.200201425s) [noise=true , address=77.90.31.17:4782 , piece_id=A4CHRLRTQLWFPXQRQFHOSP4V2ORYHQPY3MGO6GFLWCEEMN2ODN6Q, seg_pos={0 0}, upload=112449]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: rpc: dial tcp 77.90.31.17:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:35.502669385 (128h6m23.200237692s) [noise=true , address=77.90.31.16:4739 , piece_id=BMO5MBOSKYVKSELFE5NODVPQ5LJFQHQWN5N5JBDHSPCOAPS6C6KA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: rpc: dial tcp 77.90.31.16:4739: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:35.502786745 (128h6m23.200355051s) [noise=true , address=77.90.31.15:4703 , piece_id=INX7IHUCBJ7OZMS6EVPLBLWRTNHSMETAMFPX5PXJ4FDHVNSIZ5FA, seg_pos={0 0}, upload=112449]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:35.502808626 (128h6m23.200376942s) [noise=true , address=77.90.31.13:4637 , piece_id=FXR6KRLUJWJNZXJSRI72YNUE24SDWE5AT3T4WJMBMN5M2GUCJPEQ, seg_pos={0 0}, upload=112449]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: rpc: dial tcp 77.90.31.13:4637: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:35.502900549 (128h6m23.200468854s) [noise=true , address=77.90.31.19:4847 , piece_id=DRCXXVCKY653OEMAFDMDRGEJ4C6ZH5H7MIESMUXZSHXRQL5SJNQQ, seg_pos={0 0}, upload=112449]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 5.161.89.252:10562->77.90.31.19:4847: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:35:35.502918873 (128h6m23.200487179s) [seg_pos={0 0}, upload=112449]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:35:35.523116981 (128h6m23.220685297s) [upload=112449]: Done uploading segments... Dec 10 23:35:35.523132671 (128h6m23.220700977s) [upload=112449]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.02207605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................................XXX.............] duration: 610.879185ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:35:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 610.900515ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................................XX.XX..............] duration: 659.194367ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:35:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 659.210005ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................................XXX............] duration: 614.366349ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 614.439886ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................X..X.X....X.X...] duration: 446.245574ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:34:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.268657ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.022156824s trace: https://jaeger.dev.storj.io/trace/5e90ed68b8f9b812 FAIL (2025-12-10 23:34:33 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Dec 10 23:33:33.026730689 (128h4m20.724299025s) [upload=112433]: Got next segment. Inline: false Dec 10 23:33:33.026773489 (128h4m20.724341825s) [upload=112433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:33:33.026782095 (128h4m20.724350441s) [upload=112433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:33:33.049691238 (128h4m20.747259575s) [seg_pos={0 0}, upload=112433]: Begin upload segment... Dec 10 23:33:33.049744819 (128h4m20.747313155s) [seg_pos={0 0}, upload=112433]: Done begin upload segment. Dec 10 23:33:33.049786045 (128h4m20.747354381s) [noise=true , address=77.90.31.16:4739 , piece_id=KE4QNCS46AMTCUSFOX3DATEZZ2ZF5AFTBXQH7UKVCXGM3F74N37A, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.049848673 (128h4m20.747417009s) [noise=true , address=145.79.227.36:4523 , piece_id=NY5DCZV4GRHQRTLJ7C4U4UUX2ONNW5SA2BPJKN3LCND3MQHXLSUA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.050371539 (128h4m20.747939875s) [noise=true , address=145.79.227.4:4214 , piece_id=XL3J7FI2OWC46AJDHYS7DHTQLMSGQZI5BNEOXDXSWAJUQYK27H2Q, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.050832219 (128h4m20.748400556s) [noise=true , address=45.91.116.14:4212 , piece_id=QTLPURHLCCFDO2ZVY54SVFFPTA6MN27EG4G45DWRFFWXKDNPVYLQ, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.051348734 (128h4m20.748917081s) [noise=true , address=77.90.31.21:4897 , piece_id=IED6TO2MH3OJYO7V2TJOOWI56OXWZ5B7ACDFUKVL6SDB35IHMQKA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.051808544 (128h4m20.749376879s) [noise=true , address=145.79.227.38:4989 , piece_id=P7CLZGP3XDDC6LTK6XSXFROGJ2EDSBAXCR6BMSXKE3HPXBTOTINA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.052354042 (128h4m20.749922378s) [noise=true , address=77.90.31.15:4721 , piece_id=QW6SNEZORC7DJ7ZWYG7IL6NCKDOPH4ORB36WFIKRSKJQBZDP4LLQ, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.052809524 (128h4m20.750377859s) [noise=true , address=45.91.116.13:4186 , piece_id=MPM2XJEJEEF5MHVIXZAE422LWXPKMVKD66T7CPQWRRN3ZNDG3JIQ, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.053318423 (128h4m20.75088676s) [noise=true , address=5.231.73.3:4029 , piece_id=QY5YVDAKOIT4SN34MM5M4ELPRDKIDP5GOPDPR34R6Q66S6S4L4WA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.053808249 (128h4m20.751376595s) [noise=true , address=5.231.73.5:4012 , piece_id=Q446USNNUMUEHTVQFACQAAZLSMEPHG2Z76RNJFC5TTQK6HHYIUAQ, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.056267563 (128h4m20.753835899s) [noise=true , address=77.90.31.14:4677 , piece_id=K3KDWBMBNPRGIDLGZF2IYRLC5YCH4TCBO6JDVXFB4R3ODY536MOA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.056913319 (128h4m20.754481705s) [noise=true , address=45.91.116.12:4252 , piece_id=4FAH7DQRMHBYBRUSAJ4A43XYKS6N4VPYSC6ISQZTFUF2KKJDYVNA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.057538256 (128h4m20.755106593s) [noise=true , address=145.79.227.6:4272 , piece_id=YGZYCLM5X3D2BP4GGM5WKL5E2GSOUSWECSJ3ZBZMOIS5PITPSQPQ, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.057859226 (128h4m20.755427562s) [noise=true , address=45.91.116.11:4144 , piece_id=L3KZG2PJ4FKQGMRMY4FU3DZVLRBB6CP3N5QZHXBMUB3S3QWUFAHA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.058466049 (128h4m20.756034395s) [noise=true , address=77.90.31.19:4857 , piece_id=B3TNNQRWCPM7QGSBO672YET6Q34MOK3DENO6MUB3N73HAFY6LAWA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.059081690 (128h4m20.756650036s) [noise=true , address=145.79.227.44:4222 , piece_id=E3KJPG5HEFQORURQSPLLEBYRK6VE66L44N2ZVLR5OUWCQYIVVJXA, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.059656453 (128h4m20.757224799s) [noise=true , address=64.49.13.137:4502 , piece_id=EYNSO45W3MNRWZAJ5FJXQBB3C4VDD43XPM7UVJY7CQ6CFLLOTHLQ, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.059846959 (128h4m20.757415294s) [noise=true , address=5.231.73.4:4033 , piece_id=H4SKIQ72AKEBP4EJ3Y5PPBIO6VWJEFTKM76YQK2FLFDM7CRQC6EQ, seg_pos={0 0}, upload=112433]: Uploading piece... Dec 10 23:33:33.163076068 (128h4m20.860644404s) [upload=112433]: Next returned nil segment Dec 10 23:33:33.163088912 (128h4m20.860657248s) [upload=112433]: Waiting for error group managing segments... Dec 10 23:33:33.178270189 (128h4m20.875838526s) [noise=true , address=45.91.116.14:4212 , piece_id=QTLPURHLCCFDO2ZVY54SVFFPTA6MN27EG4G45DWRFFWXKDNPVYLQ, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.178624091 (128h4m20.876192418s) [noise=true , address=145.79.227.6:4272 , piece_id=YGZYCLM5X3D2BP4GGM5WKL5E2GSOUSWECSJ3ZBZMOIS5PITPSQPQ, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.179300285 (128h4m20.87686861s) [noise=true , address=145.79.227.4:4214 , piece_id=XL3J7FI2OWC46AJDHYS7DHTQLMSGQZI5BNEOXDXSWAJUQYK27H2Q, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.179860110 (128h4m20.877428436s) [noise=true , address=145.79.227.36:4523 , piece_id=NY5DCZV4GRHQRTLJ7C4U4UUX2ONNW5SA2BPJKN3LCND3MQHXLSUA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.189136749 (128h4m20.886705085s) [noise=true , address=45.91.116.13:4186 , piece_id=MPM2XJEJEEF5MHVIXZAE422LWXPKMVKD66T7CPQWRRN3ZNDG3JIQ, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.190248405 (128h4m20.887816741s) [noise=true , address=145.79.227.38:4989 , piece_id=P7CLZGP3XDDC6LTK6XSXFROGJ2EDSBAXCR6BMSXKE3HPXBTOTINA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.195010731 (128h4m20.892579068s) [noise=true , address=45.91.116.11:4144 , piece_id=L3KZG2PJ4FKQGMRMY4FU3DZVLRBB6CP3N5QZHXBMUB3S3QWUFAHA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.205425085 (128h4m20.90299342s) [noise=true , address=45.91.116.12:4252 , piece_id=4FAH7DQRMHBYBRUSAJ4A43XYKS6N4VPYSC6ISQZTFUF2KKJDYVNA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.256933049 (128h4m20.954501386s) [noise=true , address=5.231.73.4:4033 , piece_id=H4SKIQ72AKEBP4EJ3Y5PPBIO6VWJEFTKM76YQK2FLFDM7CRQC6EQ, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.282313179 (128h4m20.979881565s) [noise=true , address=5.231.73.5:4012 , piece_id=Q446USNNUMUEHTVQFACQAAZLSMEPHG2Z76RNJFC5TTQK6HHYIUAQ, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.285188870 (128h4m20.982757207s) [noise=true , address=5.231.73.3:4029 , piece_id=QY5YVDAKOIT4SN34MM5M4ELPRDKIDP5GOPDPR34R6Q66S6S4L4WA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:33.655793480 (128h4m21.353361816s) [noise=true , address=64.49.13.137:4502 , piece_id=EYNSO45W3MNRWZAJ5FJXQBB3C4VDD43XPM7UVJY7CQ6CFLLOTHLQ, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:33:34.550108153 (128h4m22.247676489s) [noise=true , address=145.79.227.44:4222 , piece_id=E3KJPG5HEFQORURQSPLLEBYRK6VE66L44N2ZVLR5OUWCQYIVVJXA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: <nil> Dec 10 23:34:33.005929738 (128h5m20.703498084s) [noise=true , address=77.90.31.21:4897 , piece_id=IED6TO2MH3OJYO7V2TJOOWI56OXWZ5B7ACDFUKVL6SDB35IHMQKA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: ecclient: upload failed (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, address:77.90.31.21:4897): protocol: rpc: dial tcp 77.90.31.21:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:33.006107380 (128h5m20.703675716s) [noise=true , address=77.90.31.19:4857 , piece_id=B3TNNQRWCPM7QGSBO672YET6Q34MOK3DENO6MUB3N73HAFY6LAWA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: ecclient: upload failed (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, address:77.90.31.19:4857): protocol: write tcp 5.161.89.252:39864->77.90.31.19:4857: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:33.006153056 (128h5m20.703721401s) [noise=true , address=77.90.31.16:4739 , piece_id=KE4QNCS46AMTCUSFOX3DATEZZ2ZF5AFTBXQH7UKVCXGM3F74N37A, seg_pos={0 0}, upload=112433]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: rpc: dial tcp 77.90.31.16:4739: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:33.006185847 (128h5m20.703754194s) [noise=true , address=77.90.31.15:4721 , piece_id=QW6SNEZORC7DJ7ZWYG7IL6NCKDOPH4ORB36WFIKRSKJQBZDP4LLQ, seg_pos={0 0}, upload=112433]: Done uploading piece. err: ecclient: upload failed (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn, address:77.90.31.15:4721): protocol: rpc: dial tcp 77.90.31.15:4721: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:33.006294039 (128h5m20.703862375s) [noise=true , address=77.90.31.14:4677 , piece_id=K3KDWBMBNPRGIDLGZF2IYRLC5YCH4TCBO6JDVXFB4R3ODY536MOA, seg_pos={0 0}, upload=112433]: Done uploading piece. err: ecclient: upload failed (node:1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, address:77.90.31.14:4677): protocol: write tcp 5.161.89.252:58612->77.90.31.14:4677: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:33.006330327 (128h5m20.703898663s) [seg_pos={0 0}, upload=112433]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:34:33.026241279 (128h5m20.723809616s) [upload=112433]: Done uploading segments... Dec 10 23:34:33.026267048 (128h5m20.723835384s) [upload=112433]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.022169538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................................XX.XX.............] duration: 503.60021ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:34:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.665652ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................X..X.X....X.X..] duration: 405.599527ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:34:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 405.621639ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration: 635.727039ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:33:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 635.746294ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 1m0.015694082s trace: https://jaeger.dev.storj.io/trace/6b7d16731d25f67c FAIL (2025-12-10 23:33:32 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== stItem_ObjectBegin Dec 10 23:32:32.946650216 (128h3m20.644218552s) [upload=112417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:32:32.971186819 (128h3m20.668755155s) [seg_pos={0 0}, upload=112417]: Begin upload segment... Dec 10 23:32:32.971224860 (128h3m20.668793207s) [seg_pos={0 0}, upload=112417]: Done begin upload segment. Dec 10 23:32:32.971274173 (128h3m20.668842509s) [noise=true , address=45.91.116.6:4983 , piece_id=PKSADI5EXLVOOIGYIW7TC4QWKUCKVDOGIMFL6ISDOKON4ZJRHEBQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.971283961 (128h3m20.668852297s) [noise=true , address=145.79.227.22:4524 , piece_id=TOEB4LQXMXEZRM6EBMBA3SSGRL7BI7HQPZMLI2MW47QZDA3PDVMQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.971820153 (128h3m20.669388489s) [noise=true , address=45.91.116.14:4204 , piece_id=LACNNVVCZ6CFIM7NQTBW6RNCYYKCMVBNFMQZMRNDB4LN3N7GOQWA, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.972295089 (128h3m20.669863436s) [noise=true , address=77.90.31.17:4791 , piece_id=NIE3FVEMCZHWX256B44SKHSZZOAGDQ2F43WLKAQZR5TEZLMNEP5Q, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.972816224 (128h3m20.670384559s) [noise=true , address=77.90.31.15:4706 , piece_id=TKMB6NW3QDPJITMX3WUJ4XVIQR65FADS2MRJ6I2SDCLCGQZ7GKVA, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.973262226 (128h3m20.670830562s) [noise=true , address=145.79.227.3:4158 , piece_id=AYMU4BOY5JLB6Q446PEQSD2F2NADW32T7YKUWVIRFHSHBMVE5H4Q, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.973726754 (128h3m20.6712951s) [noise=true , address=77.90.31.12:4605 , piece_id=EDW5OFP6U32XOIBOOZIP54ZQFLNW7YZ7U743PQJ74DS2GOEJ4AJA, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.974154693 (128h3m20.67172303s) [noise=true , address=77.90.31.19:4856 , piece_id=QABDCKIZFFY3F5KWLGMP4ZQRZ5YOX3BCJV3LMRUGZPUQLOGADCKQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.974600937 (128h3m20.672169273s) [noise=true , address=145.79.227.40:4064 , piece_id=QD4PWWJCRWKULUNM4YK3KXB5G76H43XMIS2HILD6XPTJR5ZFN3FA, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.975023486 (128h3m20.672591823s) [noise=true , address=145.79.227.44:4205 , piece_id=LU6GMTUROPOENJOJBTXE5T5NRJC5UIXDQRIQVSTNBEJLTX5MR6MA, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.975456165 (128h3m20.673024501s) [noise=true , address=5.231.73.4:4003 , piece_id=WQXRYFFJ2SMNP6M4L76RYJ7EIVNIUG62M7CUCZ4ZJGIJZ4MXUIWQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.977065300 (128h3m20.674633636s) [noise=true , address=77.90.31.14:4690 , piece_id=FUCYOE7JOKXU5OWTRQUIJF7HM2ZOTWQZSJUWTHYJHY76V5565HQQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.977449537 (128h3m20.675017874s) [noise=true , address=45.91.116.4:4945 , piece_id=7RTCVZZAVPOHFBYKERLLCQQMNH2RFMGBCEURWW2EUEAPHFGGT4DA, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.978008982 (128h3m20.675577319s) [noise=true , address=45.91.116.7:4017 , piece_id=IXTUKJBXHVR2ONYQY25WOL7IEOFBHD7FVXBZYVZJVLFPDJTQN6SQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.978136692 (128h3m20.675705028s) [noise=true , address=45.91.116.14:4220 , piece_id=JLEQ6JRWWQVZ3NSXLXQQKKLHYGYGSA5AHGNIR2O2WQIRT4PW73AQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.978707407 (128h3m20.676275753s) [noise=true , address=77.90.31.13:4649 , piece_id=WNWMZUN6UTH3QY43PEDSCAF7QQLYJPVVQY2PTN2CNWMHM7LENGSQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.979101975 (128h3m20.676670311s) [noise=true , address=45.91.116.12:4237 , piece_id=LBQZO6H5F5QKLHP7J4CQJXUW23JD4NG5GKNPEPFCFMN7UG7L6E4A, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:32.979633658 (128h3m20.677201994s) [noise=true , address=64.49.13.139:4476 , piece_id=CJLI7M6WFCOW3V4QT7ZJOTFYVXKCFAALNR33Z6L7CHADIMM2N3LQ, seg_pos={0 0}, upload=112417]: Uploading piece... Dec 10 23:32:33.075755841 (128h3m20.773324177s) [upload=112417]: Next returned nil segment Dec 10 23:32:33.075773594 (128h3m20.773341941s) [upload=112417]: Waiting for error group managing segments... Dec 10 23:32:33.085271626 (128h3m20.782839962s) [noise=true , address=45.91.116.4:4945 , piece_id=7RTCVZZAVPOHFBYKERLLCQQMNH2RFMGBCEURWW2EUEAPHFGGT4DA, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.085940626 (128h3m20.783508962s) [noise=true , address=45.91.116.12:4237 , piece_id=LBQZO6H5F5QKLHP7J4CQJXUW23JD4NG5GKNPEPFCFMN7UG7L6E4A, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.086597093 (128h3m20.784165429s) [noise=true , address=145.79.227.22:4524 , piece_id=TOEB4LQXMXEZRM6EBMBA3SSGRL7BI7HQPZMLI2MW47QZDA3PDVMQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.096116304 (128h3m20.79368464s) [noise=true , address=45.91.116.7:4017 , piece_id=IXTUKJBXHVR2ONYQY25WOL7IEOFBHD7FVXBZYVZJVLFPDJTQN6SQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.096585079 (128h3m20.794153416s) [noise=true , address=145.79.227.3:4158 , piece_id=AYMU4BOY5JLB6Q446PEQSD2F2NADW32T7YKUWVIRFHSHBMVE5H4Q, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.102209276 (128h3m20.799777611s) [noise=true , address=45.91.116.6:4983 , piece_id=PKSADI5EXLVOOIGYIW7TC4QWKUCKVDOGIMFL6ISDOKON4ZJRHEBQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.104096751 (128h3m20.801665088s) [noise=true , address=45.91.116.14:4220 , piece_id=JLEQ6JRWWQVZ3NSXLXQQKKLHYGYGSA5AHGNIR2O2WQIRT4PW73AQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.104375082 (128h3m20.801943417s) [noise=true , address=45.91.116.14:4204 , piece_id=LACNNVVCZ6CFIM7NQTBW6RNCYYKCMVBNFMQZMRNDB4LN3N7GOQWA, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.130112007 (128h3m20.827680343s) [noise=true , address=145.79.227.40:4064 , piece_id=QD4PWWJCRWKULUNM4YK3KXB5G76H43XMIS2HILD6XPTJR5ZFN3FA, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.163115957 (128h3m20.860684294s) [noise=true , address=5.231.73.4:4003 , piece_id=WQXRYFFJ2SMNP6M4L76RYJ7EIVNIUG62M7CUCZ4ZJGIJZ4MXUIWQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:33.585437783 (128h3m21.283006118s) [noise=true , address=64.49.13.139:4476 , piece_id=CJLI7M6WFCOW3V4QT7ZJOTFYVXKCFAALNR33Z6L7CHADIMM2N3LQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:32:34.025409148 (128h3m21.722977493s) [noise=true , address=145.79.227.44:4205 , piece_id=LU6GMTUROPOENJOJBTXE5T5NRJC5UIXDQRIQVSTNBEJLTX5MR6MA, seg_pos={0 0}, upload=112417]: Done uploading piece. err: <nil> Dec 10 23:33:32.932125569 (128h4m20.629693915s) [noise=true , address=77.90.31.17:4791 , piece_id=NIE3FVEMCZHWX256B44SKHSZZOAGDQ2F43WLKAQZR5TEZLMNEP5Q, seg_pos={0 0}, upload=112417]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: rpc: dial tcp 77.90.31.17:4791: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:32.932277814 (128h4m20.629846159s) [noise=true , address=77.90.31.14:4690 , piece_id=FUCYOE7JOKXU5OWTRQUIJF7HM2ZOTWQZSJUWTHYJHY76V5565HQQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:32.932298052 (128h4m20.629866388s) [noise=true , address=77.90.31.15:4706 , piece_id=TKMB6NW3QDPJITMX3WUJ4XVIQR65FADS2MRJ6I2SDCLCGQZ7GKVA, seg_pos={0 0}, upload=112417]: Done uploading piece. err: ecclient: upload failed (node:12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, address:77.90.31.15:4706): protocol: write tcp 5.161.89.252:26626->77.90.31.15:4706: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:32.932397888 (128h4m20.629966224s) [noise=true , address=77.90.31.13:4649 , piece_id=WNWMZUN6UTH3QY43PEDSCAF7QQLYJPVVQY2PTN2CNWMHM7LENGSQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:32.932517632 (128h4m20.630085979s) [noise=true , address=77.90.31.12:4605 , piece_id=EDW5OFP6U32XOIBOOZIP54ZQFLNW7YZ7U743PQJ74DS2GOEJ4AJA, seg_pos={0 0}, upload=112417]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: rpc: dial tcp 77.90.31.12:4605: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:32.932609694 (128h4m20.63017803s) [noise=true , address=77.90.31.19:4856 , piece_id=QABDCKIZFFY3F5KWLGMP4ZQRZ5YOX3BCJV3LMRUGZPUQLOGADCKQ, seg_pos={0 0}, upload=112417]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 5.161.89.252:46032->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:32.932640321 (128h4m20.630208658s) [seg_pos={0 0}, upload=112417]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:33:32.945460569 (128h4m20.643028905s) [upload=112417]: Done uploading segments... Dec 10 23:33:32.945515251 (128h4m20.643083588s) [upload=112417]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.015713037s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................X..X.X....X.X.] duration: 381.486937ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:33:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 381.521881ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................................XX.XX............] duration: 483.400186ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:33:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.414713ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X..X.X....X.X] duration: 30.000503721s trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:32:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000549006s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration: 793.645967ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:32:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 793.720117ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................................XX.XX...........] duration: 465.853751ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:32:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.879079ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 1m0.020491731s trace: https://jaeger.dev.storj.io/trace/776995f34a19fe20 FAIL (2025-12-10 23:32:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 10 23:31:26.197273176 (128h2m13.894841512s) [upload=112401]: Starting upload Dec 10 23:31:26.197345782 (128h2m13.894914127s) [upload=112401]: Done starting upload Dec 10 23:31:26.197430360 (128h2m13.894998706s) [upload=112401]: Uploading segments... Dec 10 23:31:26.197739787 (128h2m13.895308123s) [upload=112401]: Got next segment. Inline: false Dec 10 23:31:26.197787407 (128h2m13.895355742s) [upload=112401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:31:26.197793718 (128h2m13.895362055s) [upload=112401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:31:26.218102665 (128h2m13.915671001s) [seg_pos={0 0}, upload=112401]: Begin upload segment... Dec 10 23:31:26.218140816 (128h2m13.915709153s) [seg_pos={0 0}, upload=112401]: Done begin upload segment. Dec 10 23:31:26.218166605 (128h2m13.915734941s) [noise=true , address=45.91.116.4:4962 , piece_id=SMT4PP4ONZL4N2QROMP2PIGB746KMYMPITNN7ERMYG55ZNSNW7GA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.218198564 (128h2m13.915766901s) [noise=true , address=145.79.227.50:4400 , piece_id=K55W76BXE2N5TZY5ER23PIL2UMQUFGOFEQK24OSYPRVWGQE2QPQA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.218691255 (128h2m13.916259591s) [noise=true , address=45.91.116.13:4171 , piece_id=NA2C2GDWQDA2ERTKK7UZ7OBFZL2SLWUG6K46F7GQHOQV3Z2EY7CQ, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.219077516 (128h2m13.916645852s) [noise=true , address=45.91.116.10:4102 , piece_id=PFGXJ4PQKIVNX53FK2NLRBB6JIT2GD2XYEZCW3PEPQL7SMWTWASA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.219443570 (128h2m13.917011906s) [noise=true , address=5.231.73.2:4008 , piece_id=TITL4GH6EWJNW5LTB6PIY75BFLE3NXNQGVPNU7ZPADFILZAVJ77Q, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.219845761 (128h2m13.917414088s) [noise=true , address=64.49.13.169:4708 , piece_id=RFS7OFOFB6C7VWQIKLGW2POVUBFLF7RJ7J5JM5VPBSELGAFA4PTA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.220254345 (128h2m13.91782267s) [noise=true , address=77.90.31.12:4617 , piece_id=HMFJ3BVQNMYVFT2P5CASPOJ6LYNPXFPDQJZIYRWUSMM2I5UTSJYQ, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.220650835 (128h2m13.918219171s) [noise=true , address=5.231.73.3:4011 , piece_id=INLYOHSL7ETRRI23JCHSLXJISRYLQA4P5EGTICZ3JG3NNN3C2WEA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.221039651 (128h2m13.918607987s) [noise=true , address=77.90.31.11:4570 , piece_id=K4767N74HRDLPHP5M2G3WSCARPSQZKDWHR6VFG6VVAGV3V4WHK2Q, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.221416494 (128h2m13.918984831s) [noise=true , address=145.79.227.8:4342 , piece_id=7RXA2QVR7W345PPZ3BVIM2CBHH5NVVSBAMVCVMK7QZWVXETUJ2IQ, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.223699450 (128h2m13.921267785s) [noise=true , address=77.90.31.21:4905 , piece_id=QUYKC6UOROJZRXPIU5YQBAEJPU3IKJJBAQMP7EWNLTFAJBEWE6JA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.224639665 (128h2m13.922208002s) [noise=true , address=77.90.31.16:4740 , piece_id=GA3XRAMNH2GZCBUGI4FBSVY4TOIVPTSC5Y6CW3KP4CUGB3PJ4BEQ, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.224926421 (128h2m13.922494757s) [noise=true , address=77.90.31.17:4794 , piece_id=7FFYZ6KTQ3PDWXJ3OESZ27JPP4IJ6KSFTHGHN73RGPLMMIEBZ73A, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.225927872 (128h2m13.923496208s) [noise=true , address=45.91.116.12:4249 , piece_id=DIWNJ2BC5KUWNWU3AZVIK5VIMH3UXRWSS2IKG7X6LNNI4VNNMBIA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.226489451 (128h2m13.924057787s) [noise=true , address=145.79.227.18:4022 , piece_id=P2CHDCBVETVNG6ZAMN7ILOX4HITSTCFGHZK5MLDBY3TGQYTPONTQ, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.227018489 (128h2m13.924586825s) [noise=true , address=145.79.227.30:4776 , piece_id=BBR2HWGCJRQWH3GZOLZQXM5S4V6J52AP4TAQ6MDB6QNK6PJTW3VA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.227314993 (128h2m13.924883329s) [noise=true , address=145.79.227.6:4262 , piece_id=3ASWE3TVR3ZA6BP5M4L7KFBDZ2RA32SFLB5F7JHAL3F3MW4LRGNA, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.227974766 (128h2m13.925543092s) [noise=true , address=64.49.13.140:4506 , piece_id=QOTEN32J526GNEGQWXWP6YD6WEAIH6MWQJXAB6KRPDFS3Y2YXZQQ, seg_pos={0 0}, upload=112401]: Uploading piece... Dec 10 23:31:26.325603914 (128h2m14.02317225s) [upload=112401]: Next returned nil segment Dec 10 23:31:26.325616909 (128h2m14.023185254s) [upload=112401]: Waiting for error group managing segments... Dec 10 23:31:26.335824526 (128h2m14.033392842s) [noise=true , address=45.91.116.12:4249 , piece_id=DIWNJ2BC5KUWNWU3AZVIK5VIMH3UXRWSS2IKG7X6LNNI4VNNMBIA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.335890520 (128h2m14.033458856s) [noise=true , address=45.91.116.4:4962 , piece_id=SMT4PP4ONZL4N2QROMP2PIGB746KMYMPITNN7ERMYG55ZNSNW7GA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.336967942 (128h2m14.034536268s) [noise=true , address=145.79.227.18:4022 , piece_id=P2CHDCBVETVNG6ZAMN7ILOX4HITSTCFGHZK5MLDBY3TGQYTPONTQ, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.337842145 (128h2m14.035410491s) [noise=true , address=145.79.227.50:4400 , piece_id=K55W76BXE2N5TZY5ER23PIL2UMQUFGOFEQK24OSYPRVWGQE2QPQA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.338591494 (128h2m14.036159821s) [noise=true , address=145.79.227.8:4342 , piece_id=7RXA2QVR7W345PPZ3BVIM2CBHH5NVVSBAMVCVMK7QZWVXETUJ2IQ, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.348860818 (128h2m14.046429153s) [noise=true , address=45.91.116.10:4102 , piece_id=PFGXJ4PQKIVNX53FK2NLRBB6JIT2GD2XYEZCW3PEPQL7SMWTWASA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.350921236 (128h2m14.048489542s) [noise=true , address=145.79.227.6:4262 , piece_id=3ASWE3TVR3ZA6BP5M4L7KFBDZ2RA32SFLB5F7JHAL3F3MW4LRGNA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.353576766 (128h2m14.051145072s) [noise=true , address=45.91.116.13:4171 , piece_id=NA2C2GDWQDA2ERTKK7UZ7OBFZL2SLWUG6K46F7GQHOQV3Z2EY7CQ, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.412276266 (128h2m14.109844612s) [noise=true , address=5.231.73.2:4008 , piece_id=TITL4GH6EWJNW5LTB6PIY75BFLE3NXNQGVPNU7ZPADFILZAVJ77Q, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.462742915 (128h2m14.160311252s) [noise=true , address=5.231.73.3:4011 , piece_id=INLYOHSL7ETRRI23JCHSLXJISRYLQA4P5EGTICZ3JG3NNN3C2WEA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:26.803994263 (128h2m14.501562599s) [noise=true , address=64.49.13.140:4506 , piece_id=QOTEN32J526GNEGQWXWP6YD6WEAIH6MWQJXAB6KRPDFS3Y2YXZQQ, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:27.219304513 (128h2m14.916872829s) [noise=true , address=145.79.227.30:4776 , piece_id=BBR2HWGCJRQWH3GZOLZQXM5S4V6J52AP4TAQ6MDB6QNK6PJTW3VA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:31:32.033099720 (128h2m19.730668036s) [noise=true , address=64.49.13.169:4708 , piece_id=RFS7OFOFB6C7VWQIKLGW2POVUBFLF7RJ7J5JM5VPBSELGAFA4PTA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: <nil> Dec 10 23:32:26.179558930 (128h3m13.877127255s) [noise=true , address=77.90.31.12:4617 , piece_id=HMFJ3BVQNMYVFT2P5CASPOJ6LYNPXFPDQJZIYRWUSMM2I5UTSJYQ, seg_pos={0 0}, upload=112401]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:32:26.179632818 (128h3m13.877201154s) [noise=true , address=77.90.31.17:4794 , piece_id=7FFYZ6KTQ3PDWXJ3OESZ27JPP4IJ6KSFTHGHN73RGPLMMIEBZ73A, seg_pos={0 0}, upload=112401]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:26.179732233 (128h3m13.87730057s) [noise=true , address=77.90.31.16:4740 , piece_id=GA3XRAMNH2GZCBUGI4FBSVY4TOIVPTSC5Y6CW3KP4CUGB3PJ4BEQ, seg_pos={0 0}, upload=112401]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:32:26.179857116 (128h3m13.877425453s) [noise=true , address=77.90.31.21:4905 , piece_id=QUYKC6UOROJZRXPIU5YQBAEJPU3IKJJBAQMP7EWNLTFAJBEWE6JA, seg_pos={0 0}, upload=112401]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: rpc: dial tcp 77.90.31.21:4905: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 10 23:32:26.180066949 (128h3m13.877635285s) [noise=true , address=77.90.31.11:4570 , piece_id=K4767N74HRDLPHP5M2G3WSCARPSQZKDWHR6VFG6VVAGV3V4WHK2Q, seg_pos={0 0}, upload=112401]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:32:26.180093909 (128h3m13.877662246s) [seg_pos={0 0}, upload=112401]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:32:26.197621921 (128h3m13.895190246s) [upload=112401]: Done uploading segments... Dec 10 23:32:26.197645214 (128h3m13.895213551s) [upload=112401]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.020504345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................................XXX.........] duration: 634.900858ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:31:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 634.929782ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................................XX.XX..........] duration: 532.60435ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:31:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 532.621983ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 1m0.0218382s trace: https://jaeger.dev.storj.io/trace/335cdb046a6bfa8d FAIL (2025-12-10 23:31:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== id=CHOD6QG6DUOHKJTEW3UL7XTSRTIYJWOAAMPRDE4U6TWKXKQRFAUA, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.143784274 (128h1m13.84135259s) [noise=true , address=45.91.116.11:4146 , piece_id=AERZWAAJNT2WPANO7UCJG4PEXRMSBP6ZBQD4OALJ4QTUKSUUM5WA, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.144322920 (128h1m13.841891256s) [noise=true , address=77.90.31.21:4887 , piece_id=2LBMYYQD7PCDITHNGL4ASNMO5UKTJBOBUWUY27S5HIMK3TQFVSEQ, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.144769274 (128h1m13.84233761s) [noise=true , address=77.90.31.19:4873 , piece_id=N2VUB5JSYZCSSRB2KRTZ242Q3I6XQXRN5TRSU653FNL7K6ZR3HDA, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.145182967 (128h1m13.842751303s) [noise=true , address=77.90.31.17:4786 , piece_id=NDAJV4LMYHNUYQSDGH4T7MEPN2ZSDRZEEK5ZRDH6FGOGVKKHESGQ, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.145877184 (128h1m13.843445519s) [noise=true , address=145.79.227.31:4820 , piece_id=HQBBTJTQPRJEW4VLG2FIS6AVYE34WZ2JBRVHC5NPUECAZVMV2GDQ, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.146532227 (128h1m13.844100573s) [noise=true , address=45.91.116.8:4064 , piece_id=IZZZREHFHOQDBN24FMFGOG4N3ZU62WYVGCNYN5RYGHCQ4WTDGRAQ, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.146976537 (128h1m13.844544873s) [noise=true , address=145.79.227.39:4047 , piece_id=NHCQLAYREM7WW55Z6QTWKFFPO6KR7WOFF5D4D6JJIHOSYQ3LSAIA, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.147409225 (128h1m13.844977562s) [noise=true , address=64.49.13.155:4921 , piece_id=L5BLGPUHPRCYAB67PVCCOWDRQ2CTGWFT6VQCXF5XMGKLMA6XYQ5Q, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.148175247 (128h1m13.845743593s) [noise=true , address=77.90.31.13:4653 , piece_id=VV3CEMNTXH7WPB5HTRSCSI4MARWXZQPBGW44C7D5DFNKJ6MQK7ZA, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.149575191 (128h1m13.847143508s) [noise=true , address=64.49.13.169:4710 , piece_id=XFXAWFKINUHPYIUPP6UGEC2KAUTITKF7QST2BUJZC5ZP42LPTLUA, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.150797003 (128h1m13.84836531s) [noise=true , address=77.90.31.13:4641 , piece_id=SGDTTAGJVEXYS4LDSSEDZT5X7BPTNSSYDBEUSMWN6TT4KOEDD2IQ, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.151864357 (128h1m13.849432673s) [noise=true , address=145.79.227.32:4847 , piece_id=XEMQ7XQ3SPJJZLIWHSZQGWNJ244ZSBXVKCA5CLD2Q433MDLBMM6A, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.154443134 (128h1m13.852011471s) [noise=true , address=77.90.31.11:4577 , piece_id=CT46IC56NJIOD5KUG4MQHUWKXQTNARNWSTEZBE32N2L7DMHDHIQA, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.156254368 (128h1m13.853822704s) [noise=true , address=45.91.116.9:4268 , piece_id=LMOIB6FOBQWGNV7REXPKZRQKDLPTXQ34VIR6KQLAUJ5GULYCJF6Q, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.157720426 (128h1m13.855288742s) [noise=true , address=77.90.31.12:4612 , piece_id=VCNN7NJ2TMYFBVAZGQWT7MSL3LJUV42VUGDX6RLUAKFGK53CR4HQ, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.158912694 (128h1m13.856480999s) [noise=true , address=77.90.31.15:4708 , piece_id=ZAJA7MUC4RDZPGNWFPX7NP2DNW5ZQLSNLEHVWKB664TRQKDJY55A, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.160095743 (128h1m13.857664059s) [noise=true , address=5.231.73.3:4022 , piece_id=HTWWRTJTZQCUCWQJZCPXK6IHGI73ESC4MBJPHZRRSH47T4TCHZSQ, seg_pos={0 0}, upload=112387]: Uploading piece... Dec 10 23:30:26.228427104 (128h1m13.92599541s) [upload=112387]: Next returned nil segment Dec 10 23:30:26.228435820 (128h1m13.926004136s) [upload=112387]: Waiting for error group managing segments... Dec 10 23:30:26.242585352 (128h1m13.940153697s) [noise=true , address=145.79.227.39:4047 , piece_id=NHCQLAYREM7WW55Z6QTWKFFPO6KR7WOFF5D4D6JJIHOSYQ3LSAIA, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:26.244929901 (128h1m13.942498207s) [noise=true , address=145.79.227.32:4847 , piece_id=XEMQ7XQ3SPJJZLIWHSZQGWNJ244ZSBXVKCA5CLD2Q433MDLBMM6A, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:26.255997685 (128h1m13.953566021s) [noise=true , address=45.91.116.8:4064 , piece_id=IZZZREHFHOQDBN24FMFGOG4N3ZU62WYVGCNYN5RYGHCQ4WTDGRAQ, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:26.257372012 (128h1m13.954940318s) [noise=true , address=45.91.116.11:4146 , piece_id=AERZWAAJNT2WPANO7UCJG4PEXRMSBP6ZBQD4OALJ4QTUKSUUM5WA, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:26.260400980 (128h1m13.957969287s) [noise=true , address=145.79.227.31:4820 , piece_id=HQBBTJTQPRJEW4VLG2FIS6AVYE34WZ2JBRVHC5NPUECAZVMV2GDQ, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:26.260608838 (128h1m13.958177185s) [noise=true , address=145.79.227.5:4256 , piece_id=CHOD6QG6DUOHKJTEW3UL7XTSRTIYJWOAAMPRDE4U6TWKXKQRFAUA, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:26.263352003 (128h1m13.960920309s) [noise=true , address=45.91.116.9:4268 , piece_id=LMOIB6FOBQWGNV7REXPKZRQKDLPTXQ34VIR6KQLAUJ5GULYCJF6Q, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:26.345353309 (128h1m14.042921625s) [noise=true , address=5.231.73.3:4022 , piece_id=HTWWRTJTZQCUCWQJZCPXK6IHGI73ESC4MBJPHZRRSH47T4TCHZSQ, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:26.766844663 (128h1m14.464412969s) [noise=true , address=64.49.13.155:4921 , piece_id=L5BLGPUHPRCYAB67PVCCOWDRQ2CTGWFT6VQCXF5XMGKLMA6XYQ5Q, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:30:31.861155567 (128h1m19.558723873s) [noise=true , address=64.49.13.169:4710 , piece_id=XFXAWFKINUHPYIUPP6UGEC2KAUTITKF7QST2BUJZC5ZP42LPTLUA, seg_pos={0 0}, upload=112387]: Done uploading piece. err: <nil> Dec 10 23:31:26.106042771 (128h2m13.803611107s) [noise=true , address=77.90.31.17:4786 , piece_id=NDAJV4LMYHNUYQSDGH4T7MEPN2ZSDRZEEK5ZRDH6FGOGVKKHESGQ, seg_pos={0 0}, upload=112387]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: rpc: dial tcp 77.90.31.17:4786: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:26.106094979 (128h2m13.803663314s) [noise=true , address=77.90.31.21:4887 , piece_id=2LBMYYQD7PCDITHNGL4ASNMO5UKTJBOBUWUY27S5HIMK3TQFVSEQ, seg_pos={0 0}, upload=112387]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:26.106244067 (128h2m13.803812403s) [noise=true , address=77.90.31.19:4873 , piece_id=N2VUB5JSYZCSSRB2KRTZ242Q3I6XQXRN5TRSU653FNL7K6ZR3HDA, seg_pos={0 0}, upload=112387]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: write tcp 5.161.89.252:27390->77.90.31.19:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:26.106259356 (128h2m13.803827692s) [noise=true , address=77.90.31.13:4653 , piece_id=VV3CEMNTXH7WPB5HTRSCSI4MARWXZQPBGW44C7D5DFNKJ6MQK7ZA, seg_pos={0 0}, upload=112387]: Done uploading piece. err: ecclient: upload failed (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, address:77.90.31.13:4653): protocol: rpc: dial tcp 77.90.31.13:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:26.106542074 (128h2m13.80411041s) [noise=true , address=77.90.31.15:4708 , piece_id=ZAJA7MUC4RDZPGNWFPX7NP2DNW5ZQLSNLEHVWKB664TRQKDJY55A, seg_pos={0 0}, upload=112387]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: rpc: dial tcp 77.90.31.15:4708: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:26.106761563 (128h2m13.80432989s) [noise=true , address=77.90.31.13:4641 , piece_id=SGDTTAGJVEXYS4LDSSEDZT5X7BPTNSSYDBEUSMWN6TT4KOEDD2IQ, seg_pos={0 0}, upload=112387]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: rpc: dial tcp 77.90.31.13:4641: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:26.106856460 (128h2m13.804424797s) [noise=true , address=77.90.31.12:4612 , piece_id=VCNN7NJ2TMYFBVAZGQWT7MSL3LJUV42VUGDX6RLUAKFGK53CR4HQ, seg_pos={0 0}, upload=112387]: Done uploading piece. err: ecclient: upload failed (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W, address:77.90.31.12:4612): protocol: rpc: dial tcp 77.90.31.12:4612: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:26.107000409 (128h2m13.804568746s) [noise=true , address=77.90.31.11:4577 , piece_id=CT46IC56NJIOD5KUG4MQHUWKXQTNARNWSTEZBE32N2L7DMHDHIQA, seg_pos={0 0}, upload=112387]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 5.161.89.252:24806->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:26.107024785 (128h2m13.804593111s) [seg_pos={0 0}, upload=112387]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:31:26.126345856 (128h2m13.823914202s) [upload=112387]: Done uploading segments... Dec 10 23:31:26.126369340 (128h2m13.823937677s) [upload=112387]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.021851275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XXX........] duration: 614.521628ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:31:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 614.53875ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................................XX.XX.........] duration: 501.414947ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:31:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 501.435886ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X..X.X....X.] duration: 365.321198ms trace: https://jaeger.dev.storj.io/trace/6588676b197327c0 FAIL (2025-12-10 23:31:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 365.339312ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X..X.X....X] duration: 30.000597324s trace: https://jaeger.dev.storj.io/trace/6588676b197327c0 FAIL (2025-12-10 23:31:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000610569s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration: 650.605252ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:30:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 650.624388ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............................................XX.XX........] duration: 545.165002ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:30:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 545.190981ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 1m0.017526727s trace: https://jaeger.dev.storj.io/trace/188cf0dc62f7d15e FAIL (2025-12-10 23:30:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 04s) [seg_pos={0 0}, upload=112368]: Begin upload segment... Dec 10 23:29:19.848256525 (128h0m7.54582483s) [seg_pos={0 0}, upload=112368]: Done begin upload segment. Dec 10 23:29:19.848276311 (128h0m7.545844628s) [noise=true , address=45.91.116.10:4092 , piece_id=ZZ25D3DXG23OFTKAOJH3DLXVHEXMWAK3SL2PNXS2SUZDQYQWAL6Q, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.848359677 (128h0m7.545927993s) [noise=true , address=145.79.227.23:4571 , piece_id=NPXSVUXNVYI54YSTFT3INZH4XLXTPWZXADYE4NHOZJEDDU5SR5XA, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.848690565 (128h0m7.546258871s) [noise=true , address=77.90.31.21:4897 , piece_id=PNR5O2UTX7M3GOXUNPCTYLXYWFV3ECPWQOYS2MIMHEX3SNN2Y7SA, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.848956051 (128h0m7.546524357s) [noise=true , address=77.90.31.13:4659 , piece_id=DPAGK3AI3NLGCVT5U5SX6BTY5BANEKADJXWF72NQIDP7TP4FZNEQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.849198454 (128h0m7.54676677s) [noise=true , address=5.231.73.2:4034 , piece_id=YK67S2A2WMZBHNBMP7EADONBV2C5RFNQUP76C7XS64INEIDNULTQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.849444204 (128h0m7.54701252s) [noise=true , address=45.91.116.12:4259 , piece_id=LDIHG3UIXGRCIRK6GN7JVF4DMNJ47BV62LBVKUMCB55NR5L6ZURA, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.849758971 (128h0m7.547327287s) [noise=true , address=77.90.31.11:4574 , piece_id=Q2KL4VX45TA2OWGTWFYQPEEWLSTNQOVFH35Q7YY5YEHD4UKM446A, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.850124244 (128h0m7.54769255s) [noise=true , address=145.79.227.43:4182 , piece_id=KS52TE22T6FMSSH53ODG2GYDS3GPDGH5O3D2RZR3AQKCUMZYMCGQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.851695790 (128h0m7.549264106s) [noise=true , address=45.91.116.13:4177 , piece_id=SOAMBBTY54VDUS7EBRQVP77TSIHNMTZCKNUEQPGLMBFMCWOFM3TQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.852135290 (128h0m7.549703596s) [noise=true , address=64.49.13.27:4966 , piece_id=ZVTQ4RX5X2VLELV34375YB5XCXNHNIFPQA64DAMUXWUNO4EQ5VFA, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.852554643 (128h0m7.550122959s) [noise=true , address=77.90.31.19:4865 , piece_id=SYLKS3JMD3ZHQAIY7CW2CO3CGFWK4S76G4DU6Z3NOCTEFO2DBRGA, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.852653268 (128h0m7.550221574s) [noise=true , address=64.49.13.169:4707 , piece_id=N77X7TMDEFDBMFX6UGUKRZB5JIPOGQQPD7DTQPZOYEZBGNE4NW3Q, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.852981140 (128h0m7.550549446s) [noise=true , address=77.90.31.12:4617 , piece_id=ETFSF4BQXED2MF7B753V45S3QU3MVDFTIR6BCFAAMSWDHLG2MSWQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.853293704 (128h0m7.55086201s) [noise=true , address=145.79.227.29:4959 , piece_id=HCVPSNZS34INOWEOCVOIBLTHKJ3NKJ6DM7XOGEVSCFVN7EPLU5MQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.853627738 (128h0m7.551196054s) [noise=true , address=5.231.73.4:4020 , piece_id=PUPYP6P7UXIMT6YQPH54HS6N5ZKMXQKMKIKOQWE55C7XVFUZKM3Q, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.854626194 (128h0m7.55219451s) [noise=true , address=77.90.31.17:4802 , piece_id=3HHU3Y3TSS2VLXIT6RNCHMSQLPBDTHSWIPVSFMNXPKO3GOYCOXKQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.855617265 (128h0m7.553185581s) [noise=true , address=77.90.31.18:4819 , piece_id=R4UGY2D2H24SZCWEINZA5AAJYDTAFC7H63TK6ZGZOAXWORC6JJLQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.856628463 (128h0m7.554196769s) [noise=true , address=64.49.13.140:4508 , piece_id=Z7RELA5CAWD6VHELCHX5ZL66ZBM47NKQUDHITXFN33ZFGBB5H5OQ, seg_pos={0 0}, upload=112368]: Uploading piece... Dec 10 23:29:19.916838844 (128h0m7.614407161s) [upload=112368]: Next returned nil segment Dec 10 23:29:19.916852701 (128h0m7.614421017s) [upload=112368]: Waiting for error group managing segments... Dec 10 23:29:19.924726930 (128h0m7.622295246s) [noise=true , address=45.91.116.13:4177 , piece_id=SOAMBBTY54VDUS7EBRQVP77TSIHNMTZCKNUEQPGLMBFMCWOFM3TQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:19.928003280 (128h0m7.625571586s) [noise=true , address=145.79.227.29:4959 , piece_id=HCVPSNZS34INOWEOCVOIBLTHKJ3NKJ6DM7XOGEVSCFVN7EPLU5MQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:19.933694651 (128h0m7.631262967s) [noise=true , address=145.79.227.23:4571 , piece_id=NPXSVUXNVYI54YSTFT3INZH4XLXTPWZXADYE4NHOZJEDDU5SR5XA, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:19.934606144 (128h0m7.632174461s) [noise=true , address=45.91.116.10:4092 , piece_id=ZZ25D3DXG23OFTKAOJH3DLXVHEXMWAK3SL2PNXS2SUZDQYQWAL6Q, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:19.938621625 (128h0m7.636189941s) [noise=true , address=145.79.227.43:4182 , piece_id=KS52TE22T6FMSSH53ODG2GYDS3GPDGH5O3D2RZR3AQKCUMZYMCGQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:19.940315659 (128h0m7.637883975s) [noise=true , address=45.91.116.12:4259 , piece_id=LDIHG3UIXGRCIRK6GN7JVF4DMNJ47BV62LBVKUMCB55NR5L6ZURA, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:20.055678915 (128h0m7.753247231s) [noise=true , address=5.231.73.4:4020 , piece_id=PUPYP6P7UXIMT6YQPH54HS6N5ZKMXQKMKIKOQWE55C7XVFUZKM3Q, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:20.139106203 (128h0m7.836674509s) [noise=true , address=5.231.73.2:4034 , piece_id=YK67S2A2WMZBHNBMP7EADONBV2C5RFNQUP76C7XS64INEIDNULTQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:20.430795601 (128h0m8.128363928s) [noise=true , address=64.49.13.27:4966 , piece_id=ZVTQ4RX5X2VLELV34375YB5XCXNHNIFPQA64DAMUXWUNO4EQ5VFA, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:20.431083920 (128h0m8.128652216s) [noise=true , address=64.49.13.140:4508 , piece_id=Z7RELA5CAWD6VHELCHX5ZL66ZBM47NKQUDHITXFN33ZFGBB5H5OQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:29:25.657034604 (128h0m13.35460292s) [noise=true , address=64.49.13.169:4707 , piece_id=N77X7TMDEFDBMFX6UGUKRZB5JIPOGQQPD7DTQPZOYEZBGNE4NW3Q, seg_pos={0 0}, upload=112368]: Done uploading piece. err: <nil> Dec 10 23:30:19.807132303 (128h1m7.504700609s) [noise=true , address=77.90.31.17:4802 , piece_id=3HHU3Y3TSS2VLXIT6RNCHMSQLPBDTHSWIPVSFMNXPKO3GOYCOXKQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: ecclient: upload failed (node:12hiNtWkLwWMYr8rPN9ukEKQou5Eko2fD9ScXsqBKk5oX5wxgHy, address:77.90.31.17:4802): protocol: rpc: dial tcp 77.90.31.17:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:19.807272866 (128h1m7.504841172s) [noise=true , address=77.90.31.11:4574 , piece_id=Q2KL4VX45TA2OWGTWFYQPEEWLSTNQOVFH35Q7YY5YEHD4UKM446A, seg_pos={0 0}, upload=112368]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:19.807394463 (128h1m7.504962769s) [noise=true , address=77.90.31.19:4865 , piece_id=SYLKS3JMD3ZHQAIY7CW2CO3CGFWK4S76G4DU6Z3NOCTEFO2DBRGA, seg_pos={0 0}, upload=112368]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: rpc: dial tcp 77.90.31.19:4865: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:19.807547719 (128h1m7.505116025s) [noise=true , address=77.90.31.21:4897 , piece_id=PNR5O2UTX7M3GOXUNPCTYLXYWFV3ECPWQOYS2MIMHEX3SNN2Y7SA, seg_pos={0 0}, upload=112368]: Done uploading piece. err: ecclient: upload failed (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, address:77.90.31.21:4897): protocol: rpc: dial tcp 77.90.31.21:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:19.807603062 (128h1m7.505171398s) [noise=true , address=77.90.31.13:4659 , piece_id=DPAGK3AI3NLGCVT5U5SX6BTY5BANEKADJXWF72NQIDP7TP4FZNEQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: write tcp 5.161.89.252:28424->77.90.31.13:4659: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:19.807635593 (128h1m7.505203899s) [noise=true , address=77.90.31.12:4617 , piece_id=ETFSF4BQXED2MF7B753V45S3QU3MVDFTIR6BCFAAMSWDHLG2MSWQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: write tcp 5.161.89.252:11112->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:19.807765105 (128h1m7.505333411s) [noise=true , address=77.90.31.18:4819 , piece_id=R4UGY2D2H24SZCWEINZA5AAJYDTAFC7H63TK6ZGZOAXWORC6JJLQ, seg_pos={0 0}, upload=112368]: Done uploading piece. err: ecclient: upload failed (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, address:77.90.31.18:4819): protocol: rpc: dial tcp 77.90.31.18:4819: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:19.807787246 (128h1m7.505355552s) [seg_pos={0 0}, upload=112368]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:30:19.821872877 (128h1m7.519441224s) [upload=112368]: Done uploading segments... Dec 10 23:30:19.821901892 (128h1m7.519470228s) [upload=112368]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.017541133s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X..X.X....] duration: 388.563634ms trace: https://jaeger.dev.storj.io/trace/454218cee1644749 FAIL (2025-12-10 23:29:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 388.576949ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XXX......] duration: 659.668422ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:29:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 659.68372ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................................................XX.XX.......] duration: 732.42543ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:29:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 732.440176ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 1m0.036157637s trace: https://jaeger.dev.storj.io/trace/4fdfad13c79e28c2 FAIL (2025-12-10 23:29:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== chRequestItem_ObjectBegin Dec 10 23:28:19.725355573 (127h59m7.422923879s) [upload=112353]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:28:19.746426573 (127h59m7.443994889s) [seg_pos={0 0}, upload=112353]: Begin upload segment... Dec 10 23:28:19.746506663 (127h59m7.444074979s) [seg_pos={0 0}, upload=112353]: Done begin upload segment. Dec 10 23:28:19.746532481 (127h59m7.444100797s) [noise=true , address=145.79.227.16:4937 , piece_id=LZ7O5E25JMSYD6BRSFAYYKGTYWVLBTYWA4WVVNBTISD7NZU3XHZQ, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.746541107 (127h59m7.444109423s) [noise=true , address=77.90.31.13:4635 , piece_id=P2E3GDEJRXXEFZQ6LQB6MMWIHIQDHXBYCYEHDOTZVTVW5WDIFBCA, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.746580010 (127h59m7.444148316s) [noise=true , address=145.79.227.44:4206 , piece_id=MPFBUTZGX2OC7RPF435XHZPGUXGMQQRQU7KZ2L43PUH27JI5OCAQ, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.746678313 (127h59m7.444246629s) [noise=true , address=77.90.31.17:4775 , piece_id=HZDCJPDAXKMDKHSKKRVSNIVVKU7UCZURFJFSWQZY33RHAXVJYRFQ, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.746968746 (127h59m7.444537062s) [noise=true , address=45.91.116.12:4257 , piece_id=HDLRSOT5YIBXGDTSB6776CIH4L5A4Y4YL3T4MBEUJCTPZ45PD75Q, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.747099179 (127h59m7.444667496s) [noise=true , address=45.91.116.5:4327 , piece_id=AMGHX3JFUEYMSD22P2NLLHNU6QJANVM5OQE77NR3EGEGWEZUIC4Q, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.747481574 (127h59m7.44504988s) [noise=true , address=45.91.116.13:4161 , piece_id=MXICWLOJMZJSJ7IHUQFHZFZT3XBK7YBXE5BDGHEBV5QUSXFRULBA, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.747725389 (127h59m7.445293695s) [noise=true , address=77.90.31.14:4685 , piece_id=INRTMBFZ7AXDEA3UXKPAVOYRAKV3BBF2EJWYPC4ULS2N4EUWKYNQ, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.747877433 (127h59m7.445445749s) [noise=true , address=145.79.227.13:4826 , piece_id=ZSZLLLUQ5UHBG3KXBCJL5JBDXT2FPY624VTT7GSPSYYX4HIQM3VA, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.747991166 (127h59m7.445559482s) [noise=true , address=77.90.31.18:4809 , piece_id=XLDQ6HFVP5KARKGM2DZ6HI3W2KBCNBVAXFH2AJDSG5YKIJRC4XSQ, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.748230623 (127h59m7.445798939s) [noise=true , address=145.79.227.50:4378 , piece_id=M3FSX5FMX4PHDM3OW3CVYUDTNBDIX2SKYLKD46FBRV3AWQQ5R7JQ, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.748332544 (127h59m7.44590086s) [noise=true , address=145.79.227.4:4198 , piece_id=MPXINS3EXPTMG75OTRMBOZQ6DQLXI6FH4THV2TIFDBFTEBDLTLHA, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.748730747 (127h59m7.446299063s) [noise=true , address=45.91.116.14:4209 , piece_id=KCIXFRIWITM4UWT56G3RAZIFYW5H5OEGL4ESFYMGPBJRI6XE26MA, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.749176620 (127h59m7.446744926s) [noise=true , address=77.90.31.21:4912 , piece_id=4I4BCTPNOST3ZCC5RDHCJRK7UWSHM32YRZY6BO2IMPHQTMLKJHTA, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.749974310 (127h59m7.447542626s) [noise=true , address=145.79.227.19:4092 , piece_id=BLIB7BAENHUUMBRF7CSQ2TFPU7WVERT26VEAQOCF4JWPI4REMQRA, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.750237561 (127h59m7.447805877s) [noise=true , address=77.90.31.12:4616 , piece_id=M3Q6W26X5NEFONUDE7SE7WGETKK4HJH5SLWZC5RTKIJXCXWEP4WQ, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.750508117 (127h59m7.448076433s) [noise=true , address=145.79.227.30:4777 , piece_id=UTMSKW2O7GPG5UKIK3IC4YOM4M4G2U7SXNFFQGY3EPP6GHHJRSRQ, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.750765859 (127h59m7.448334165s) [noise=true , address=45.91.116.11:4123 , piece_id=WY57R26RYV4ZASS5GAU6LKLDMEMFN4CHQ6HN5XSFYN6IRZIIJUFA, seg_pos={0 0}, upload=112353]: Uploading piece... Dec 10 23:28:19.833314648 (127h59m7.530882984s) [upload=112353]: Next returned nil segment Dec 10 23:28:19.833327522 (127h59m7.530895857s) [upload=112353]: Waiting for error group managing segments... Dec 10 23:28:19.842119785 (127h59m7.539688132s) [noise=true , address=45.91.116.14:4209 , piece_id=KCIXFRIWITM4UWT56G3RAZIFYW5H5OEGL4ESFYMGPBJRI6XE26MA, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:19.844125563 (127h59m7.541693879s) [noise=true , address=145.79.227.19:4092 , piece_id=BLIB7BAENHUUMBRF7CSQ2TFPU7WVERT26VEAQOCF4JWPI4REMQRA, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:19.844183500 (127h59m7.541751816s) [noise=true , address=45.91.116.13:4161 , piece_id=MXICWLOJMZJSJ7IHUQFHZFZT3XBK7YBXE5BDGHEBV5QUSXFRULBA, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:19.844514168 (127h59m7.542082484s) [noise=true , address=145.79.227.13:4826 , piece_id=ZSZLLLUQ5UHBG3KXBCJL5JBDXT2FPY624VTT7GSPSYYX4HIQM3VA, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:19.844543713 (127h59m7.54211202s) [noise=true , address=145.79.227.50:4378 , piece_id=M3FSX5FMX4PHDM3OW3CVYUDTNBDIX2SKYLKD46FBRV3AWQQ5R7JQ, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:19.846848128 (127h59m7.544416444s) [noise=true , address=145.79.227.4:4198 , piece_id=MPXINS3EXPTMG75OTRMBOZQ6DQLXI6FH4THV2TIFDBFTEBDLTLHA, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:19.850476966 (127h59m7.548045272s) [noise=true , address=45.91.116.5:4327 , piece_id=AMGHX3JFUEYMSD22P2NLLHNU6QJANVM5OQE77NR3EGEGWEZUIC4Q, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:19.852254918 (127h59m7.549823224s) [noise=true , address=45.91.116.11:4123 , piece_id=WY57R26RYV4ZASS5GAU6LKLDMEMFN4CHQ6HN5XSFYN6IRZIIJUFA, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:19.859655822 (127h59m7.557224158s) [noise=true , address=45.91.116.12:4257 , piece_id=HDLRSOT5YIBXGDTSB6776CIH4L5A4Y4YL3T4MBEUJCTPZ45PD75Q, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:20.819604340 (127h59m8.517172676s) [noise=true , address=145.79.227.30:4777 , piece_id=UTMSKW2O7GPG5UKIK3IC4YOM4M4G2U7SXNFFQGY3EPP6GHHJRSRQ, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:20.846416956 (127h59m8.543985292s) [noise=true , address=145.79.227.44:4206 , piece_id=MPFBUTZGX2OC7RPF435XHZPGUXGMQQRQU7KZ2L43PUH27JI5OCAQ, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:28:20.904324344 (127h59m8.60189268s) [noise=true , address=145.79.227.16:4937 , piece_id=LZ7O5E25JMSYD6BRSFAYYKGTYWVLBTYWA4WVVNBTISD7NZU3XHZQ, seg_pos={0 0}, upload=112353]: Done uploading piece. err: <nil> Dec 10 23:29:19.713575495 (128h0m7.411143811s) [noise=true , address=77.90.31.12:4616 , piece_id=M3Q6W26X5NEFONUDE7SE7WGETKK4HJH5SLWZC5RTKIJXCXWEP4WQ, seg_pos={0 0}, upload=112353]: Done uploading piece. err: ecclient: upload failed (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, address:77.90.31.12:4616): protocol: rpc: dial tcp 77.90.31.12:4616: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:19.713615910 (128h0m7.411184226s) [noise=true , address=77.90.31.14:4685 , piece_id=INRTMBFZ7AXDEA3UXKPAVOYRAKV3BBF2EJWYPC4ULS2N4EUWKYNQ, seg_pos={0 0}, upload=112353]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:19.713739682 (128h0m7.411307988s) [noise=true , address=77.90.31.17:4775 , piece_id=HZDCJPDAXKMDKHSKKRVSNIVVKU7UCZURFJFSWQZY33RHAXVJYRFQ, seg_pos={0 0}, upload=112353]: Done uploading piece. err: ecclient: upload failed (node:127VKvwi82kYfGZ6Wcv8BinaNUQBMQsgKLKNNLht5JPsBLN7ovt, address:77.90.31.17:4775): protocol: rpc: dial tcp 77.90.31.17:4775: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:19.713832846 (128h0m7.411401162s) [noise=true , address=77.90.31.18:4809 , piece_id=XLDQ6HFVP5KARKGM2DZ6HI3W2KBCNBVAXFH2AJDSG5YKIJRC4XSQ, seg_pos={0 0}, upload=112353]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:19.714017391 (128h0m7.411585707s) [noise=true , address=77.90.31.13:4635 , piece_id=P2E3GDEJRXXEFZQ6LQB6MMWIHIQDHXBYCYEHDOTZVTVW5WDIFBCA, seg_pos={0 0}, upload=112353]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: write tcp 5.161.89.252:48198->77.90.31.13:4635: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:19.714038890 (128h0m7.411607206s) [noise=true , address=77.90.31.21:4912 , piece_id=4I4BCTPNOST3ZCC5RDHCJRK7UWSHM32YRZY6BO2IMPHQTMLKJHTA, seg_pos={0 0}, upload=112353]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 5.161.89.252:52432->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:19.714060170 (128h0m7.411628476s) [seg_pos={0 0}, upload=112353]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:29:19.748519409 (128h0m7.446087715s) [upload=112353]: Done uploading segments... Dec 10 23:29:19.748543253 (128h0m7.44611157s) [upload=112353]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.036168627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration: 900.580909ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:28:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 900.599584ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X..X.X...] duration: 412.668495ms trace: https://jaeger.dev.storj.io/trace/454218cee1644749 FAIL (2025-12-10 23:28:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 412.690456ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................................................XX.XX......] duration: 514.201243ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:28:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 514.221741ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X..X.X..] duration: 409.362346ms trace: https://jaeger.dev.storj.io/trace/454218cee1644749 FAIL (2025-12-10 23:28:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 409.3804ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.018621738s trace: https://jaeger.dev.storj.io/trace/3fba58ae710e8bb3 FAIL (2025-12-10 23:28:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Dec 10 23:27:19.681550102 (127h58m7.379118418s) [seg_pos={0 0}, upload=112343]: Begin upload segment... Dec 10 23:27:19.681581230 (127h58m7.379149546s) [seg_pos={0 0}, upload=112343]: Done begin upload segment. Dec 10 23:27:19.681619011 (127h58m7.379187318s) [noise=true , address=145.79.227.26:4682 , piece_id=RF3XDAZOFXKVSTF5HQJBAMPLQPI2LATYWWE6RBDDL3U3MKCHLIOQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.681627767 (127h58m7.379196084s) [noise=true , address=45.91.116.7:4025 , piece_id=EPINVLL3ITDXM4P25CBYVBPUCEHB7MGU272POY6VGYTS4Y3CKC5Q, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682040538 (127h58m7.379608845s) [noise=true , address=45.91.116.10:4107 , piece_id=CCJPHA7YKCYA6LGEO4MFJEM444HJQZYLONSSAL6VWBPRNS5F7J3Q, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682123012 (127h58m7.379691318s) [noise=true , address=77.90.31.19:4865 , piece_id=2AHBZRXC7TWWE54B7NMSCHQCWDOHQLOOBSPTJ5QKIHI7TUD6GL6A, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682366848 (127h58m7.379935164s) [noise=true , address=64.49.13.169:4706 , piece_id=UPH2ZKJZQWM4WBMW4HV5ZEXKAM4SIVAHBD3X54IGQJV3KU5ZDAQQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682411221 (127h58m7.379979527s) [noise=true , address=45.91.116.11:4127 , piece_id=53RQOBTUYP2MA2MXSA6BKFN2E5ELO5NOLOBA5H5BRGQKPUSCLZLQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682706613 (127h58m7.380274929s) [noise=true , address=145.79.227.27:4710 , piece_id=7FBDRLN2SFMS4YYIZMKZFJZNCBL3NK5JP3OEZ7AM5YF2JEGK7QZA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682799836 (127h58m7.380368142s) [noise=true , address=77.90.31.17:4779 , piece_id=5X4LOQORHXU27QY2IHMUPKTRLDNQOMC2E3KTSAKVDUCZPKZ5ZK3A, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.682952782 (127h58m7.380521098s) [noise=true , address=77.90.31.12:4617 , piece_id=EHWCQDIJJ2TC6MUAUHVQLUQTANA5G322AVTSKWUET3AOQX6ERFMQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683155231 (127h58m7.380723537s) [noise=true , address=77.90.31.15:4726 , piece_id=6J2T4CQLERLF4UFH3R3O2OHZPAUHOI52WLOGC3G7LQPRV5OCAF6A, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683197590 (127h58m7.380765895s) [noise=true , address=45.91.116.13:4183 , piece_id=YGPJWSOHPWYSOVKP6UMZS2Z26Y6TKINSE6JWHDIACF55DO4UMRZA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683431507 (127h58m7.380999813s) [noise=true , address=77.90.31.11:4585 , piece_id=VG4YW4PLQ6JN6WN3E63MGP56JQYCP5ODPCHFSFZLXGISQLPWRZYA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683539318 (127h58m7.381107624s) [noise=true , address=5.231.73.4:4003 , piece_id=63OP26YEX2NS2PXPHRZOSHCV75E4GTKIHHHOUA66FAFPHFGDFWOA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683675372 (127h58m7.381243679s) [noise=true , address=145.79.227.6:4269 , piece_id=PA4VKRYYLDN5IVZXPCK5YYXGJ4BUXL2UZXTINAKZHUS5764RAWBA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683886857 (127h58m7.381455163s) [noise=true , address=145.79.227.30:4771 , piece_id=EF5UEFUVPXMDRBRYP45V2BRVKOB2ADIYRDJ664ZHKIOBBHB3V7EA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.683971396 (127h58m7.381539712s) [noise=true , address=77.90.31.16:4765 , piece_id=3OGZDEK2N4CNSQWBJVMCODS4QPENAY4ICDOXWKPOJOVGTDPFSNFA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.684226221 (127h58m7.381794527s) [noise=true , address=45.91.116.5:4323 , piece_id=EFCTT36ROZZ7IOLQOBC52ESK774YBSNQHQEA22JZIQZLKZ6TEQUA, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.684487549 (127h58m7.382055855s) [noise=true , address=45.91.116.6:4996 , piece_id=5L2BAN466IC76CX44RFQCPTPFH4CETR4TTFHEU5JN3UIEKKCSSRQ, seg_pos={0 0}, upload=112343]: Uploading piece... Dec 10 23:27:19.755764983 (127h58m7.453333288s) [upload=112343]: Next returned nil segment Dec 10 23:27:19.755773819 (127h58m7.453342125s) [upload=112343]: Waiting for error group managing segments... Dec 10 23:27:19.764164843 (127h58m7.461733159s) [noise=true , address=45.91.116.11:4127 , piece_id=53RQOBTUYP2MA2MXSA6BKFN2E5ELO5NOLOBA5H5BRGQKPUSCLZLQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.764652745 (127h58m7.462221051s) [noise=true , address=45.91.116.6:4996 , piece_id=5L2BAN466IC76CX44RFQCPTPFH4CETR4TTFHEU5JN3UIEKKCSSRQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.765027465 (127h58m7.462595771s) [noise=true , address=45.91.116.13:4183 , piece_id=YGPJWSOHPWYSOVKP6UMZS2Z26Y6TKINSE6JWHDIACF55DO4UMRZA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.769236768 (127h58m7.466805074s) [noise=true , address=145.79.227.27:4710 , piece_id=7FBDRLN2SFMS4YYIZMKZFJZNCBL3NK5JP3OEZ7AM5YF2JEGK7QZA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.771386334 (127h58m7.46895464s) [noise=true , address=45.91.116.5:4323 , piece_id=EFCTT36ROZZ7IOLQOBC52ESK774YBSNQHQEA22JZIQZLKZ6TEQUA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.777824189 (127h58m7.475392495s) [noise=true , address=45.91.116.10:4107 , piece_id=CCJPHA7YKCYA6LGEO4MFJEM444HJQZYLONSSAL6VWBPRNS5F7J3Q, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.782284811 (127h58m7.479853127s) [noise=true , address=145.79.227.26:4682 , piece_id=RF3XDAZOFXKVSTF5HQJBAMPLQPI2LATYWWE6RBDDL3U3MKCHLIOQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.783013341 (127h58m7.480581648s) [noise=true , address=45.91.116.7:4025 , piece_id=EPINVLL3ITDXM4P25CBYVBPUCEHB7MGU272POY6VGYTS4Y3CKC5Q, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.787877468 (127h58m7.485445784s) [noise=true , address=145.79.227.6:4269 , piece_id=PA4VKRYYLDN5IVZXPCK5YYXGJ4BUXL2UZXTINAKZHUS5764RAWBA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:19.863831357 (127h58m7.561399663s) [noise=true , address=5.231.73.4:4003 , piece_id=63OP26YEX2NS2PXPHRZOSHCV75E4GTKIHHHOUA66FAFPHFGDFWOA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:20.708076114 (127h58m8.40564445s) [noise=true , address=145.79.227.30:4771 , piece_id=EF5UEFUVPXMDRBRYP45V2BRVKOB2ADIYRDJ664ZHKIOBBHB3V7EA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:27:32.149149796 (127h58m19.846718112s) [noise=true , address=64.49.13.169:4706 , piece_id=UPH2ZKJZQWM4WBMW4HV5ZEXKAM4SIVAHBD3X54IGQJV3KU5ZDAQQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: <nil> Dec 10 23:28:19.644676307 (127h59m7.342244613s) [noise=true , address=77.90.31.16:4765 , piece_id=3OGZDEK2N4CNSQWBJVMCODS4QPENAY4ICDOXWKPOJOVGTDPFSNFA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK, address:77.90.31.16:4765): protocol: write tcp 5.161.89.252:8882->77.90.31.16:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.644762338 (127h59m7.342330644s) [noise=true , address=77.90.31.12:4617 , piece_id=EHWCQDIJJ2TC6MUAUHVQLUQTANA5G322AVTSKWUET3AOQX6ERFMQ, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: write tcp 5.161.89.252:8210->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.644829012 (127h59m7.342397328s) [noise=true , address=77.90.31.17:4779 , piece_id=5X4LOQORHXU27QY2IHMUPKTRLDNQOMC2E3KTSAKVDUCZPKZ5ZK3A, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN, address:77.90.31.17:4779): protocol: rpc: dial tcp 77.90.31.17:4779: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.644888714 (127h59m7.34245702s) [noise=true , address=77.90.31.19:4865 , piece_id=2AHBZRXC7TWWE54B7NMSCHQCWDOHQLOOBSPTJ5QKIHI7TUD6GL6A, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: write tcp 5.161.89.252:11072->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.645022775 (127h59m7.342591091s) [noise=true , address=77.90.31.11:4585 , piece_id=VG4YW4PLQ6JN6WN3E63MGP56JQYCP5ODPCHFSFZLXGISQLPWRZYA, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: write tcp 5.161.89.252:39502->77.90.31.11:4585: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:28:19.645194445 (127h59m7.342762761s) [noise=true , address=77.90.31.15:4726 , piece_id=6J2T4CQLERLF4UFH3R3O2OHZPAUHOI52WLOGC3G7LQPRV5OCAF6A, seg_pos={0 0}, upload=112343]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: write tcp 5.161.89.252:27054->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 10 23:28:19.645233558 (127h59m7.342801864s) [seg_pos={0 0}, upload=112343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:28:19.661401509 (127h59m7.358969825s) [upload=112343]: Done uploading segments... Dec 10 23:28:19.661423810 (127h59m7.358992117s) [upload=112343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.018632889s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration: 606.773803ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:27:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 606.789211ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................................................XX.XX.....] duration: 479.202596ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:27:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.24685ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X..X.X.] duration: 378.797349ms trace: https://jaeger.dev.storj.io/trace/454218cee1644749 FAIL (2025-12-10 23:27:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 378.812979ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.023032945s trace: https://jaeger.dev.storj.io/trace/5c4d409e29f36c6c FAIL (2025-12-10 23:27:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== uestItem_ObjectBegin Dec 10 23:26:19.579841936 (127h57m7.277410273s) [upload=112324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:26:19.600543667 (127h57m7.298111972s) [seg_pos={0 0}, upload=112324]: Begin upload segment... Dec 10 23:26:19.600584852 (127h57m7.298153168s) [seg_pos={0 0}, upload=112324]: Done begin upload segment. Dec 10 23:26:19.600615400 (127h57m7.298183706s) [noise=true , address=77.90.31.17:4798 , piece_id=33MCTJ7D3RGLRFPIAE3335YJ5JBHQE4BV5OLT2KDXT3PUQKQO77Q, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.600654854 (127h57m7.29822319s) [noise=true , address=145.79.227.36:4533 , piece_id=ZHC42MNF33ATFJL76WZ333T5E2BWGJX4RMN4HATOJPXSY73NRRMQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601166870 (127h57m7.298735186s) [noise=true , address=77.90.31.16:4759 , piece_id=54IGT3HOBEGB3B4DEP43XQCZGJXS7NXRQWZIQ5Z4LLVWZV7EPPOQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601181628 (127h57m7.298749963s) [noise=true , address=5.231.73.2:4012 , piece_id=AJHGFXD66LC4JLSERUMDQITR6EVRENVJH6VFGLOQUMD7LHGDK5QA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601430613 (127h57m7.298998919s) [noise=true , address=145.79.227.28:4728 , piece_id=GZGVXWTHWEKWGTQAAIQZGRXTGWAL4RJZKXIF7VQCSY3BUA5KGMRA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601637520 (127h57m7.299205855s) [noise=true , address=45.91.116.13:4161 , piece_id=VQYSSMPZFZSBMZFMN65SQMSRQRHWES6HD3TDBRXCXY7LRSHZZ4LQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601692822 (127h57m7.299261138s) [noise=true , address=45.91.116.11:4136 , piece_id=ZWIB2CLSZH67GMO6J2CC72YTANT7CSWY4JBRA6OBTIFGEPQPSDQQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.601929084 (127h57m7.2994974s) [noise=true , address=64.49.13.26:4953 , piece_id=H7N53TT5IN7UNYVRCDFPHR3KJUDVQKPZNFDL4ZGLNBCGK4MJT3QA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602040532 (127h57m7.299608878s) [noise=true , address=145.79.227.11:4752 , piece_id=BVXXW7W5P4IPAZDUV4YDCEKNAVCPBI7AB2DUHE3SFHY4JA4GDTFA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602177889 (127h57m7.299746205s) [noise=true , address=45.91.116.6:4983 , piece_id=NBDW5F7M2PPB2Z3PK5EVGWVVZIQNFH3G5BQOUJIZGCPUWQUMMVGA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602418538 (127h57m7.299986854s) [noise=true , address=145.79.227.21:4484 , piece_id=RMQ4P3IIOTS4TFU7RNKRDBAV4YLG53RY6WDH54TJL3A43IV6A7RA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602467991 (127h57m7.300036326s) [noise=true , address=64.49.13.154:4929 , piece_id=H5WIUBNV4YDNC7LWBGUZETUW746VWVVVD6R7HQLAMGSKUCNNT5HA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602682811 (127h57m7.300251117s) [noise=true , address=77.90.31.14:4695 , piece_id=KCUPUEFVMFMPCTMEQVCQB4R65CJYTPRFF2H4KYW43T7FIVZVZZZA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602870663 (127h57m7.300439009s) [noise=true , address=64.49.13.155:4913 , piece_id=SRJNPBJNTPWWSJSFVDJXIJRPOB4ZKMVJAWO5Q43P5IJULZGSB6CQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.602921428 (127h57m7.300489744s) [noise=true , address=45.91.116.8:4062 , piece_id=UVAZCGUHD43CLY47C6SQBFQSGGRESKD7YLVBRMTCFEKTG46UDTYQ, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.603267955 (127h57m7.300836291s) [noise=true , address=77.90.31.11:4578 , piece_id=X4XP3W4N7JFGMNWJZTQDGJJALNY45BA4AH52OY2T6NVJGRFLVTIA, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.603797844 (127h57m7.301366181s) [noise=true , address=77.90.31.19:4872 , piece_id=M5ZK7E6JF6QOAB5YIOUQTQFRKQNV52BPAS5CWIHY6IWH5ZHZNE2Q, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.604327254 (127h57m7.30189556s) [noise=true , address=77.90.31.21:4912 , piece_id=RKF4UR2UB2D7UDTF2LIQPPOOONKSQWDUTASMJQNBN4CEPQKNKD6Q, seg_pos={0 0}, upload=112324]: Uploading piece... Dec 10 23:26:19.691888123 (127h57m7.38945643s) [upload=112324]: Next returned nil segment Dec 10 23:26:19.691899936 (127h57m7.389468242s) [upload=112324]: Waiting for error group managing segments... Dec 10 23:26:19.702414046 (127h57m7.399982382s) [noise=true , address=45.91.116.8:4062 , piece_id=UVAZCGUHD43CLY47C6SQBFQSGGRESKD7YLVBRMTCFEKTG46UDTYQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.703047660 (127h57m7.400615966s) [noise=true , address=45.91.116.11:4136 , piece_id=ZWIB2CLSZH67GMO6J2CC72YTANT7CSWY4JBRA6OBTIFGEPQPSDQQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.703598299 (127h57m7.401166605s) [noise=true , address=145.79.227.21:4484 , piece_id=RMQ4P3IIOTS4TFU7RNKRDBAV4YLG53RY6WDH54TJL3A43IV6A7RA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.704055682 (127h57m7.401623988s) [noise=true , address=145.79.227.36:4533 , piece_id=ZHC42MNF33ATFJL76WZ333T5E2BWGJX4RMN4HATOJPXSY73NRRMQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.704260034 (127h57m7.401828371s) [noise=true , address=45.91.116.6:4983 , piece_id=NBDW5F7M2PPB2Z3PK5EVGWVVZIQNFH3G5BQOUJIZGCPUWQUMMVGA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.706483678 (127h57m7.404051984s) [noise=true , address=145.79.227.11:4752 , piece_id=BVXXW7W5P4IPAZDUV4YDCEKNAVCPBI7AB2DUHE3SFHY4JA4GDTFA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.719169484 (127h57m7.41673783s) [noise=true , address=45.91.116.13:4161 , piece_id=VQYSSMPZFZSBMZFMN65SQMSRQRHWES6HD3TDBRXCXY7LRSHZZ4LQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.720469854 (127h57m7.41803816s) [noise=true , address=145.79.227.28:4728 , piece_id=GZGVXWTHWEKWGTQAAIQZGRXTGWAL4RJZKXIF7VQCSY3BUA5KGMRA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:19.837660411 (127h57m7.535228718s) [noise=true , address=5.231.73.2:4012 , piece_id=AJHGFXD66LC4JLSERUMDQITR6EVRENVJH6VFGLOQUMD7LHGDK5QA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:20.173210389 (127h57m7.870778705s) [noise=true , address=64.49.13.154:4929 , piece_id=H5WIUBNV4YDNC7LWBGUZETUW746VWVVVD6R7HQLAMGSKUCNNT5HA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:20.206097370 (127h57m7.903665686s) [noise=true , address=64.49.13.26:4953 , piece_id=H7N53TT5IN7UNYVRCDFPHR3KJUDVQKPZNFDL4ZGLNBCGK4MJT3QA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:26:20.209856212 (127h57m7.907424528s) [noise=true , address=64.49.13.155:4913 , piece_id=SRJNPBJNTPWWSJSFVDJXIJRPOB4ZKMVJAWO5Q43P5IJULZGSB6CQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: <nil> Dec 10 23:27:19.564917327 (127h58m7.262485632s) [noise=true , address=77.90.31.17:4798 , piece_id=33MCTJ7D3RGLRFPIAE3335YJ5JBHQE4BV5OLT2KDXT3PUQKQO77Q, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.564932515 (127h58m7.26250082s) [noise=true , address=77.90.31.21:4912 , piece_id=RKF4UR2UB2D7UDTF2LIQPPOOONKSQWDUTASMJQNBN4CEPQKNKD6Q, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: rpc: dial tcp 77.90.31.21:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565120857 (127h58m7.262689163s) [noise=true , address=77.90.31.16:4759 , piece_id=54IGT3HOBEGB3B4DEP43XQCZGJXS7NXRQWZIQ5Z4LLVWZV7EPPOQ, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: rpc: dial tcp 77.90.31.16:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565269625 (127h58m7.262837931s) [noise=true , address=77.90.31.14:4695 , piece_id=KCUPUEFVMFMPCTMEQVCQB4R65CJYTPRFF2H4KYW43T7FIVZVZZZA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 5.161.89.252:8202->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565496899 (127h58m7.263065205s) [noise=true , address=77.90.31.19:4872 , piece_id=M5ZK7E6JF6QOAB5YIOUQTQFRKQNV52BPAS5CWIHY6IWH5ZHZNE2Q, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL, address:77.90.31.19:4872): protocol: write tcp 5.161.89.252:14264->77.90.31.19:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565567061 (127h58m7.263135366s) [noise=true , address=77.90.31.11:4578 , piece_id=X4XP3W4N7JFGMNWJZTQDGJJALNY45BA4AH52OY2T6NVJGRFLVTIA, seg_pos={0 0}, upload=112324]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: rpc: dial tcp 77.90.31.11:4578: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:19.565602506 (127h58m7.263170822s) [seg_pos={0 0}, upload=112324]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:27:19.586667165 (127h58m7.284235471s) [upload=112324]: Done uploading segments... Dec 10 23:27:19.586686340 (127h58m7.284254647s) [upload=112324]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.023043905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X..X.X] duration: 30.001346416s trace: https://jaeger.dev.storj.io/trace/454218cee1644749 FAIL (2025-12-10 23:27:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001362527s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................................................XX.XX....] duration: 477.085862ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:27:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.115928ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)