Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 548.543995ms FAIL (2025-08-19 17:01:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 548.618896ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 794.270544ms FAIL (2025-08-19 17:01:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 794.292195ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....X...................] duration: 646.528705ms FAIL (2025-08-19 17:01:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 646.558731ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 554.194736ms FAIL (2025-08-19 17:00:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 554.2177ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.094450163s FAIL (2025-08-19 17:00:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 1.094478706s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 889.887082ms FAIL (2025-08-19 17:00:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 889.908695ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 170.365159ms FAIL (2025-08-19 17:00:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 170.38666ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....X..................] duration: 505.771911ms FAIL (2025-08-19 17:00:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 505.800604ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 389.818795ms FAIL (2025-08-19 17:00:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 389.838761ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 1.082814783s FAIL (2025-08-19 16:59:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.082831935s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 172.181753ms FAIL (2025-08-19 16:59:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 172.200388ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X....X.................] duration: 981.495623ms FAIL (2025-08-19 16:59:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 981.522623ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 211.21471ms FAIL (2025-08-19 16:59:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 211.233465ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 566.422771ms FAIL (2025-08-19 16:59:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 566.446967ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X....X................] duration: 361.975125ms FAIL (2025-08-19 16:59:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 361.997857ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 618.209789ms FAIL (2025-08-19 16:58:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 618.234596ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 164.853492ms FAIL (2025-08-19 16:58:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 164.88532ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X....X...............] duration: 419.425222ms FAIL (2025-08-19 16:58:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 419.445319ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X....X..............] duration: 573.390486ms FAIL (2025-08-19 16:58:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 573.417467ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 560.54395ms FAIL (2025-08-19 16:58:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 560.567454ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 218.633633ms FAIL (2025-08-19 16:58:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 218.66912ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 686.044564ms FAIL (2025-08-19 16:57:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 686.105589ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X....X.............] duration: 341.878246ms FAIL (2025-08-19 16:57:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 341.901951ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 543.630551ms FAIL (2025-08-19 16:57:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 543.652793ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 373.1211ms FAIL (2025-08-19 16:56:58 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 373.143482ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 426.049933ms FAIL (2025-08-19 16:56:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.071693ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X....X............] duration: 497.880378ms FAIL (2025-08-19 16:56:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 497.905264ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 154.275527ms FAIL (2025-08-19 16:56:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 154.298651ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 837.935366ms FAIL (2025-08-19 16:56:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 837.961877ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 362.396531ms FAIL (2025-08-19 16:55:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 362.413944ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 487.260339ms FAIL (2025-08-19 16:55:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.281068ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X....X..........] duration: 406.976875ms FAIL (2025-08-19 16:55:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 406.998317ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 497.725817ms FAIL (2025-08-19 16:55:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 497.739713ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 511.377104ms FAIL (2025-08-19 16:55:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 511.398224ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration: 442.001107ms FAIL (2025-08-19 16:55:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 442.022077ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 231.062515ms FAIL (2025-08-19 16:54:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 231.083084ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 961.363725ms FAIL (2025-08-19 16:54:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 961.391387ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 385.227555ms FAIL (2025-08-19 16:54:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 385.244587ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 137.381763ms FAIL (2025-08-19 16:54:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 137.40197ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X....X........] duration: 336.670316ms FAIL (2025-08-19 16:54:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 336.697288ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 272.434417ms FAIL (2025-08-19 16:53:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 272.461308ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 374.041359ms FAIL (2025-08-19 16:53:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.063371ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 183.797397ms FAIL (2025-08-19 16:53:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 183.82551ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X....X.......] duration: 385.305672ms FAIL (2025-08-19 16:53:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 385.329186ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X....X......] duration: 344.31783ms FAIL (2025-08-19 16:52:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 344.344961ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 723.82667ms FAIL (2025-08-19 16:52:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 723.84269ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 475.225126ms FAIL (2025-08-19 16:52:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 475.245072ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 1.021314154s FAIL (2025-08-19 16:52:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.021330506s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X....X.....] duration: 330.732766ms FAIL (2025-08-19 16:52:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 330.752623ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 254.826435ms FAIL (2025-08-19 16:52:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 254.845379ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)