Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 512.172199ms FAIL (2025-08-19 17:18:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 512.197877ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 1.005772814s FAIL (2025-08-19 17:18:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 1.005794635s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 11.102427835s FAIL (2025-08-19 17:18:28 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.102451089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 465.529443ms FAIL (2025-08-19 17:18:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 465.543069ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 333.727232ms FAIL (2025-08-19 17:18:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 333.756097ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 379.607214ms FAIL (2025-08-19 17:17:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 379.647941ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X............................................X...........X.] duration: 147.685075ms FAIL (2025-08-19 17:17:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 147.738967ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 384.709268ms FAIL (2025-08-19 17:17:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 384.722544ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 550.547635ms FAIL (2025-08-19 17:17:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 550.574243ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration: 357.850661ms FAIL (2025-08-19 17:17:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 357.883804ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X....X..........................................] duration: 372.945445ms FAIL (2025-08-19 17:16:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 372.963699ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 418.055898ms FAIL (2025-08-19 17:16:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 418.071427ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X............................................X...........X] duration: 87.299942ms FAIL (2025-08-19 17:16:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:16:50.148435305 (263h50m30.879702956s) [upload=274393]: Starting upload Aug 19 17:16:50.148525875 (263h50m30.879793526s) [upload=274393]: Done starting upload Aug 19 17:16:50.148615374 (263h50m30.879883023s) [upload=274393]: Uploading segments... Aug 19 17:16:50.148667010 (263h50m30.879934661s) [upload=274393]: Got next segment. Inline: true Aug 19 17:16:50.148674014 (263h50m30.879941663s) [upload=274393]: Waiting for error group managing segments... Aug 19 17:16:50.148720332 (263h50m30.879987981s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:16:50.148725801 (263h50m30.879993452s) [upload=274393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:16:50.148729298 (263h50m30.879996948s) [upload=274393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:16:50.221372078 (263h50m30.952639728s) [upload=274393]: Done uploading segments... Aug 19 17:16:50.221390462 (263h50m30.952658113s) [upload=274393]: Upload finished. err: metaclient: Too Many Requests Duration:* 87.322383ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 475.167033ms FAIL (2025-08-19 17:16:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 475.202369ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X............................................X...........] duration: 202.954857ms FAIL (2025-08-19 17:16:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 202.975217ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 422.648297ms FAIL (2025-08-19 17:16:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 422.679296ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 724.276245ms FAIL (2025-08-19 17:16:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 724.293316ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 624.74869ms FAIL (2025-08-19 17:15:25 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 624.770259ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X............................................X..........] duration: 439.311317ms FAIL (2025-08-19 17:15:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 439.334161ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 431.559091ms FAIL (2025-08-19 17:15:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 431.578338ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 369.404541ms FAIL (2025-08-19 17:15:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 369.421152ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X............................................X.........] duration: 333.815279ms FAIL (2025-08-19 17:15:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 333.83078ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 354.70891ms FAIL (2025-08-19 17:14:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 354.747261ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 549.329195ms FAIL (2025-08-19 17:14:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 549.361105ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X............................................X........] duration: 254.877662ms FAIL (2025-08-19 17:14:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 254.90275ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 395.788677ms FAIL (2025-08-19 17:14:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 395.807412ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 409.762687ms FAIL (2025-08-19 17:14:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 409.782484ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 435.302356ms FAIL (2025-08-19 17:14:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 435.327273ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration: 344.234586ms FAIL (2025-08-19 17:14:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 344.260635ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 388.591363ms FAIL (2025-08-19 17:13:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 388.623914ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 537.102487ms FAIL (2025-08-19 17:13:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 537.129348ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 464.686399ms FAIL (2025-08-19 17:13:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 464.790715ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 376.78401ms FAIL (2025-08-19 17:13:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 376.846608ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X............................................X.......] duration: 188.170838ms FAIL (2025-08-19 17:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 188.201876ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 479.980718ms FAIL (2025-08-19 17:13:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 480.012348ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X............................................X......] duration: 275.957321ms FAIL (2025-08-19 17:13:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 275.98278ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 409.136929ms FAIL (2025-08-19 17:12:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 409.161024ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 479.366643ms FAIL (2025-08-19 17:12:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 479.392672ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 371.794051ms FAIL (2025-08-19 17:12:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 371.85145ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X............................................X.....] duration: 391.899089ms FAIL (2025-08-19 17:12:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 391.917193ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 483.01026ms FAIL (2025-08-19 17:12:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.032661ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 350.130353ms FAIL (2025-08-19 17:12:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 350.151232ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X............................................X....] duration: 224.01536ms FAIL (2025-08-19 17:12:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 224.033986ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 448.1682ms FAIL (2025-08-19 17:11:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.191864ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X............................................X...] duration: 223.005525ms FAIL (2025-08-19 17:11:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 223.028728ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....X..................................] duration: 367.901857ms FAIL (2025-08-19 17:11:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 367.920922ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 410.35921ms FAIL (2025-08-19 17:11:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 410.384017ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X............................................X..] duration: 198.076922ms FAIL (2025-08-19 17:11:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 198.10167ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X....X.................................] duration: 595.730092ms FAIL (2025-08-19 17:11:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 595.752093ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 417.24284ms FAIL (2025-08-19 17:11:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 417.269801ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)