Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 373.1211ms FAIL (2025-08-19 16:56:58 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 373.143482ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 426.049933ms FAIL (2025-08-19 16:56:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.071693ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X....X............] duration: 497.880378ms FAIL (2025-08-19 16:56:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 497.905264ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 154.275527ms FAIL (2025-08-19 16:56:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 154.298651ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 837.935366ms FAIL (2025-08-19 16:56:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 837.961877ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 362.396531ms FAIL (2025-08-19 16:55:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 362.413944ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 487.260339ms FAIL (2025-08-19 16:55:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.281068ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X....X..........] duration: 406.976875ms FAIL (2025-08-19 16:55:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 406.998317ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 497.725817ms FAIL (2025-08-19 16:55:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 497.739713ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 511.377104ms FAIL (2025-08-19 16:55:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 511.398224ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration: 442.001107ms FAIL (2025-08-19 16:55:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 442.022077ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 231.062515ms FAIL (2025-08-19 16:54:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 231.083084ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 961.363725ms FAIL (2025-08-19 16:54:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 961.391387ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 385.227555ms FAIL (2025-08-19 16:54:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 385.244587ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 137.381763ms FAIL (2025-08-19 16:54:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 137.40197ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X....X........] duration: 336.670316ms FAIL (2025-08-19 16:54:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 336.697288ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 272.434417ms FAIL (2025-08-19 16:53:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 272.461308ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 374.041359ms FAIL (2025-08-19 16:53:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.063371ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 183.797397ms FAIL (2025-08-19 16:53:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 183.82551ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X....X.......] duration: 385.305672ms FAIL (2025-08-19 16:53:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 385.329186ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X....X......] duration: 344.31783ms FAIL (2025-08-19 16:52:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 344.344961ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 723.82667ms FAIL (2025-08-19 16:52:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 723.84269ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 475.225126ms FAIL (2025-08-19 16:52:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 475.245072ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 1.021314154s FAIL (2025-08-19 16:52:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.021330506s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X....X.....] duration: 330.732766ms FAIL (2025-08-19 16:52:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 330.752623ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 254.826435ms FAIL (2025-08-19 16:52:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 254.845379ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X....X....] duration: 393.510952ms FAIL (2025-08-19 16:51:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 393.534156ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 181.891044ms FAIL (2025-08-19 16:51:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 181.912594ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 413.55601ms FAIL (2025-08-19 16:51:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 413.581026ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 487.987834ms FAIL (2025-08-19 16:51:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 488.019325ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X....X...] duration: 349.582436ms FAIL (2025-08-19 16:50:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 349.606421ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 462.188853ms FAIL (2025-08-19 16:50:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 462.211426ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 466.100135ms FAIL (2025-08-19 16:50:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 466.215741ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 705.38992ms FAIL (2025-08-19 16:50:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 705.410759ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X....X..] duration: 2.997162657s FAIL (2025-08-19 16:49:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 2.997214856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 359.458244ms FAIL (2025-08-19 16:49:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 359.48279ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 15.643880101s FAIL (2025-08-19 16:49:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 15.643895793s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 373.989121ms FAIL (2025-08-19 16:49:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 374.00436ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.014533209s FAIL (2025-08-19 16:49:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.014592151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 191.92834ms FAIL (2025-08-19 16:49:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 191.956103ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 136.481885ms FAIL (2025-08-19 16:48:49 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 136.50031ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.030016684s FAIL (2025-08-19 16:48:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 1.030036521s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 390.229733ms FAIL (2025-08-19 16:48:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 390.24889ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 250.598766ms FAIL (2025-08-19 16:48:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 250.620427ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 456.768528ms FAIL (2025-08-19 16:48:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.855912ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.66116254s FAIL (2025-08-19 16:47:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 19 16:44:48.753149172 (263h18m29.484416802s) [upload=273826]: Uploading segments... Aug 19 16:44:48.753579761 (263h18m29.484847391s) [upload=273826]: Got next segment. Inline: false Aug 19 16:44:48.753610659 (263h18m29.484878279s) [upload=273826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:44:48.753614366 (263h18m29.484881996s) [upload=273826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:44:48.776292219 (263h18m29.50755988s) [seg_pos={0 0}, upload=273826]: Begin upload segment... Aug 19 16:44:48.776356179 (263h18m29.50762384s) [seg_pos={0 0}, upload=273826]: Done begin upload segment. Aug 19 16:44:48.776364475 (263h18m29.507632105s) [noise=true , address=77.90.31.10:4520 , piece_id=PR6IHW3DFBEOTHBZJVO2RM46NKNMNM2G4GBQQKGJMFH5BQL7BRZQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.776402286 (263h18m29.507669936s) [noise=true , address=104.36.82.6:4875 , piece_id=SJB6ANZ32RJPFJYM7I36HWDZUWSSLLVCXMLKUKTJRFNOZRTZJ46A, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.776913657 (263h18m29.508181287s) [noise=true , address=104.36.82.5:4840 , piece_id=NZ3VFI4DZQQS7TF2OBDPZBRE7OTPWHBBLVVMJPKIVVVTVHFMQOAA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.777442051 (263h18m29.508709711s) [noise=true , address=77.90.31.12:4589 , piece_id=POP4IFY2JQBLW5LUZ2VCK6ZKEO2P3CFCTOFYAQ2P3BPWVTWQZ2VA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.778228729 (263h18m29.50949638s) [noise=true , address=45.91.116.9:4262 , piece_id=A4CTCU62Z5QQOLXLB6JKTXO33PKU4ANMKUZHEFONKLAFMDSK3DAQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.778662134 (263h18m29.509929755s) [noise=true , address=104.36.82.11:4114 , piece_id=IQD3QRIDQI4OT2Z2YEP3DWWPXQDUIKVK3TVFCNIQ4FKMQ3Z3BLYQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.779186459 (263h18m29.51045409s) [noise=true , address=104.36.82.15:4257 , piece_id=CHU2NGNZC7VGUDWIV64O3JLLZZIRB2EM5Y7SBHEIPBHFAN5SHYJA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.779565813 (263h18m29.510833433s) [noise=true , address=104.36.82.10:4079 , piece_id=BPUL2JEMPHWRHJDACX3F5CLUID54OXKYV62KPAO5K4M4VOBG7TXQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.779902366 (263h18m29.511169986s) [noise=true , address=45.91.116.12:4226 , piece_id=3GLTCHKXDLEEVYH7O5DEOAVN35AX533VO3EMV42PMBFMOQIH5OOQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.780279685 (263h18m29.511547305s) [noise=true , address=104.36.82.2:4731 , piece_id=6LHTJWC2C5NUMEPAYG4JYK7GWJCUVJXDRT7BBGVWUAONANVZU3AA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.780665371 (263h18m29.511933001s) [noise=true , address=104.36.82.8:4972 , piece_id=2LCZYTUGBLRSBIE5QNDBR5YI4VUCAMHDXGKWYVPKJANDZVWNNWDA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.781153227 (263h18m29.512420857s) [noise=true , address=45.91.116.14:4191 , piece_id=N2XA6NOODNSGFBDYZKNFSYNCJMJP5KNEN74CKWPZXITNTHYO6VLQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.781681570 (263h18m29.512949231s) [noise=true , address=77.90.31.11:4555 , piece_id=JM7ALUT5Q4DWKT362YOQNMEK5MVNUSF4AQ2Y6X3DCHM2WLKOZDYA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.782413166 (263h18m29.513680826s) [noise=true , address=104.36.82.7:4909 , piece_id=CMTR44T5EYEQI5JKFCHYXG5QJSRYPC7E7ZNWHWN46UHBN2355E4A, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.783228208 (263h18m29.514495858s) [noise=true , address=45.91.116.10:4081 , piece_id=TXBONEN6HP7NR7ZEGZGQC7O24TRCU6VV6Z52FFT4VKUGQDFDLYUA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.783983739 (263h18m29.515251389s) [noise=true , address=104.36.82.3:4767 , piece_id=3K74Y4HAUTGNB3FQACZ6AJG2SCNLWV6ESGC6B25X6UVMHGRW6XKQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.811423724 (263h18m29.542691364s) [noise=true , address=104.36.82.15:4257 , piece_id=CHU2NGNZC7VGUDWIV64O3JLLZZIRB2EM5Y7SBHEIPBHFAN5SHYJA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 5.161.89.252:47900->104.36.82.15:4257: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:44:48.943227685 (263h18m29.674495376s) [upload=273826]: Next returned nil segment Aug 19 16:44:48.943250098 (263h18m29.674517747s) [upload=273826]: Waiting for error group managing segments... Aug 19 16:44:48.956597390 (263h18m29.687865039s) [noise=true , address=104.36.82.3:4767 , piece_id=3K74Y4HAUTGNB3FQACZ6AJG2SCNLWV6ESGC6B25X6UVMHGRW6XKQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.957292937 (263h18m29.688560587s) [noise=true , address=104.36.82.5:4840 , piece_id=NZ3VFI4DZQQS7TF2OBDPZBRE7OTPWHBBLVVMJPKIVVVTVHFMQOAA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.958567583 (263h18m29.689835233s) [noise=true , address=104.36.82.8:4972 , piece_id=2LCZYTUGBLRSBIE5QNDBR5YI4VUCAMHDXGKWYVPKJANDZVWNNWDA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.958715411 (263h18m29.689983061s) [noise=true , address=104.36.82.7:4909 , piece_id=CMTR44T5EYEQI5JKFCHYXG5QJSRYPC7E7ZNWHWN46UHBN2355E4A, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.959593561 (263h18m29.690861212s) [noise=true , address=104.36.82.6:4875 , piece_id=SJB6ANZ32RJPFJYM7I36HWDZUWSSLLVCXMLKUKTJRFNOZRTZJ46A, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.960205873 (263h18m29.691473523s) [noise=true , address=104.36.82.10:4079 , piece_id=BPUL2JEMPHWRHJDACX3F5CLUID54OXKYV62KPAO5K4M4VOBG7TXQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.960573964 (263h18m29.691841604s) [noise=true , address=104.36.82.2:4731 , piece_id=6LHTJWC2C5NUMEPAYG4JYK7GWJCUVJXDRT7BBGVWUAONANVZU3AA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.961045952 (263h18m29.692313602s) [noise=true , address=104.36.82.11:4114 , piece_id=IQD3QRIDQI4OT2Z2YEP3DWWPXQDUIKVK3TVFCNIQ4FKMQ3Z3BLYQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:45:18.735158964 (263h18m59.466426614s) [noise=true , address=45.91.116.12:4226 , piece_id=3GLTCHKXDLEEVYH7O5DEOAVN35AX533VO3EMV42PMBFMOQIH5OOQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735231871 (263h18m59.46649952s) [noise=true , address=77.90.31.10:4520 , piece_id=PR6IHW3DFBEOTHBZJVO2RM46NKNMNM2G4GBQQKGJMFH5BQL7BRZQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735244925 (263h18m59.466512575s) [noise=true , address=77.90.31.11:4555 , piece_id=JM7ALUT5Q4DWKT362YOQNMEK5MVNUSF4AQ2Y6X3DCHM2WLKOZDYA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735289909 (263h18m59.46655756s) [noise=true , address=77.90.31.12:4589 , piece_id=POP4IFY2JQBLW5LUZ2VCK6ZKEO2P3CFCTOFYAQ2P3BPWVTWQZ2VA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735406830 (263h18m59.46667448s) [noise=true , address=45.91.116.10:4081 , piece_id=TXBONEN6HP7NR7ZEGZGQC7O24TRCU6VV6Z52FFT4VKUGQDFDLYUA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 5.161.89.252:33248->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735457554 (263h18m59.466725205s) [noise=true , address=45.91.116.14:4191 , piece_id=N2XA6NOODNSGFBDYZKNFSYNCJMJP5KNEN74CKWPZXITNTHYO6VLQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 5.161.89.252:63752->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735514722 (263h18m59.466782372s) [noise=true , address=45.91.116.9:4262 , piece_id=A4CTCU62Z5QQOLXLB6JKTXO33PKU4ANMKUZHEFONKLAFMDSK3DAQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 5.161.89.252:36920->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735560308 (263h18m59.466827959s) [seg_pos={0 0}, upload=273826]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:18.796277230 (263h18m59.527544891s) [upload=273826]: Done uploading segments... Aug 19 16:45:18.796312307 (263h18m59.527579957s) [upload=273826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.661179311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 200.387613ms FAIL (2025-08-19 16:47:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 200.411749ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 729.799688ms FAIL (2025-08-19 16:47:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 729.820709ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 244.574507ms FAIL (2025-08-19 16:47:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 244.605595ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 367.169881ms FAIL (2025-08-19 16:47:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 19 16:44:48.753149172 (263h18m29.484416802s) [upload=273826]: Uploading segments... Aug 19 16:44:48.753579761 (263h18m29.484847391s) [upload=273826]: Got next segment. Inline: false Aug 19 16:44:48.753610659 (263h18m29.484878279s) [upload=273826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:44:48.753614366 (263h18m29.484881996s) [upload=273826]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:44:48.776292219 (263h18m29.50755988s) [seg_pos={0 0}, upload=273826]: Begin upload segment... Aug 19 16:44:48.776356179 (263h18m29.50762384s) [seg_pos={0 0}, upload=273826]: Done begin upload segment. Aug 19 16:44:48.776364475 (263h18m29.507632105s) [noise=true , address=77.90.31.10:4520 , piece_id=PR6IHW3DFBEOTHBZJVO2RM46NKNMNM2G4GBQQKGJMFH5BQL7BRZQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.776402286 (263h18m29.507669936s) [noise=true , address=104.36.82.6:4875 , piece_id=SJB6ANZ32RJPFJYM7I36HWDZUWSSLLVCXMLKUKTJRFNOZRTZJ46A, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.776913657 (263h18m29.508181287s) [noise=true , address=104.36.82.5:4840 , piece_id=NZ3VFI4DZQQS7TF2OBDPZBRE7OTPWHBBLVVMJPKIVVVTVHFMQOAA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.777442051 (263h18m29.508709711s) [noise=true , address=77.90.31.12:4589 , piece_id=POP4IFY2JQBLW5LUZ2VCK6ZKEO2P3CFCTOFYAQ2P3BPWVTWQZ2VA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.778228729 (263h18m29.50949638s) [noise=true , address=45.91.116.9:4262 , piece_id=A4CTCU62Z5QQOLXLB6JKTXO33PKU4ANMKUZHEFONKLAFMDSK3DAQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.778662134 (263h18m29.509929755s) [noise=true , address=104.36.82.11:4114 , piece_id=IQD3QRIDQI4OT2Z2YEP3DWWPXQDUIKVK3TVFCNIQ4FKMQ3Z3BLYQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.779186459 (263h18m29.51045409s) [noise=true , address=104.36.82.15:4257 , piece_id=CHU2NGNZC7VGUDWIV64O3JLLZZIRB2EM5Y7SBHEIPBHFAN5SHYJA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.779565813 (263h18m29.510833433s) [noise=true , address=104.36.82.10:4079 , piece_id=BPUL2JEMPHWRHJDACX3F5CLUID54OXKYV62KPAO5K4M4VOBG7TXQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.779902366 (263h18m29.511169986s) [noise=true , address=45.91.116.12:4226 , piece_id=3GLTCHKXDLEEVYH7O5DEOAVN35AX533VO3EMV42PMBFMOQIH5OOQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.780279685 (263h18m29.511547305s) [noise=true , address=104.36.82.2:4731 , piece_id=6LHTJWC2C5NUMEPAYG4JYK7GWJCUVJXDRT7BBGVWUAONANVZU3AA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.780665371 (263h18m29.511933001s) [noise=true , address=104.36.82.8:4972 , piece_id=2LCZYTUGBLRSBIE5QNDBR5YI4VUCAMHDXGKWYVPKJANDZVWNNWDA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.781153227 (263h18m29.512420857s) [noise=true , address=45.91.116.14:4191 , piece_id=N2XA6NOODNSGFBDYZKNFSYNCJMJP5KNEN74CKWPZXITNTHYO6VLQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.781681570 (263h18m29.512949231s) [noise=true , address=77.90.31.11:4555 , piece_id=JM7ALUT5Q4DWKT362YOQNMEK5MVNUSF4AQ2Y6X3DCHM2WLKOZDYA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.782413166 (263h18m29.513680826s) [noise=true , address=104.36.82.7:4909 , piece_id=CMTR44T5EYEQI5JKFCHYXG5QJSRYPC7E7ZNWHWN46UHBN2355E4A, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.783228208 (263h18m29.514495858s) [noise=true , address=45.91.116.10:4081 , piece_id=TXBONEN6HP7NR7ZEGZGQC7O24TRCU6VV6Z52FFT4VKUGQDFDLYUA, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.783983739 (263h18m29.515251389s) [noise=true , address=104.36.82.3:4767 , piece_id=3K74Y4HAUTGNB3FQACZ6AJG2SCNLWV6ESGC6B25X6UVMHGRW6XKQ, seg_pos={0 0}, upload=273826]: Uploading piece... Aug 19 16:44:48.811423724 (263h18m29.542691364s) [noise=true , address=104.36.82.15:4257 , piece_id=CHU2NGNZC7VGUDWIV64O3JLLZZIRB2EM5Y7SBHEIPBHFAN5SHYJA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 5.161.89.252:47900->104.36.82.15:4257: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:44:48.943227685 (263h18m29.674495376s) [upload=273826]: Next returned nil segment Aug 19 16:44:48.943250098 (263h18m29.674517747s) [upload=273826]: Waiting for error group managing segments... Aug 19 16:44:48.956597390 (263h18m29.687865039s) [noise=true , address=104.36.82.3:4767 , piece_id=3K74Y4HAUTGNB3FQACZ6AJG2SCNLWV6ESGC6B25X6UVMHGRW6XKQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.957292937 (263h18m29.688560587s) [noise=true , address=104.36.82.5:4840 , piece_id=NZ3VFI4DZQQS7TF2OBDPZBRE7OTPWHBBLVVMJPKIVVVTVHFMQOAA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.958567583 (263h18m29.689835233s) [noise=true , address=104.36.82.8:4972 , piece_id=2LCZYTUGBLRSBIE5QNDBR5YI4VUCAMHDXGKWYVPKJANDZVWNNWDA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.958715411 (263h18m29.689983061s) [noise=true , address=104.36.82.7:4909 , piece_id=CMTR44T5EYEQI5JKFCHYXG5QJSRYPC7E7ZNWHWN46UHBN2355E4A, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.959593561 (263h18m29.690861212s) [noise=true , address=104.36.82.6:4875 , piece_id=SJB6ANZ32RJPFJYM7I36HWDZUWSSLLVCXMLKUKTJRFNOZRTZJ46A, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.960205873 (263h18m29.691473523s) [noise=true , address=104.36.82.10:4079 , piece_id=BPUL2JEMPHWRHJDACX3F5CLUID54OXKYV62KPAO5K4M4VOBG7TXQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.960573964 (263h18m29.691841604s) [noise=true , address=104.36.82.2:4731 , piece_id=6LHTJWC2C5NUMEPAYG4JYK7GWJCUVJXDRT7BBGVWUAONANVZU3AA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:44:48.961045952 (263h18m29.692313602s) [noise=true , address=104.36.82.11:4114 , piece_id=IQD3QRIDQI4OT2Z2YEP3DWWPXQDUIKVK3TVFCNIQ4FKMQ3Z3BLYQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: <nil> Aug 19 16:45:18.735158964 (263h18m59.466426614s) [noise=true , address=45.91.116.12:4226 , piece_id=3GLTCHKXDLEEVYH7O5DEOAVN35AX533VO3EMV42PMBFMOQIH5OOQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735231871 (263h18m59.46649952s) [noise=true , address=77.90.31.10:4520 , piece_id=PR6IHW3DFBEOTHBZJVO2RM46NKNMNM2G4GBQQKGJMFH5BQL7BRZQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735244925 (263h18m59.466512575s) [noise=true , address=77.90.31.11:4555 , piece_id=JM7ALUT5Q4DWKT362YOQNMEK5MVNUSF4AQ2Y6X3DCHM2WLKOZDYA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735289909 (263h18m59.46655756s) [noise=true , address=77.90.31.12:4589 , piece_id=POP4IFY2JQBLW5LUZ2VCK6ZKEO2P3CFCTOFYAQ2P3BPWVTWQZ2VA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735406830 (263h18m59.46667448s) [noise=true , address=45.91.116.10:4081 , piece_id=TXBONEN6HP7NR7ZEGZGQC7O24TRCU6VV6Z52FFT4VKUGQDFDLYUA, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 5.161.89.252:33248->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735457554 (263h18m59.466725205s) [noise=true , address=45.91.116.14:4191 , piece_id=N2XA6NOODNSGFBDYZKNFSYNCJMJP5KNEN74CKWPZXITNTHYO6VLQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 5.161.89.252:63752->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735514722 (263h18m59.466782372s) [noise=true , address=45.91.116.9:4262 , piece_id=A4CTCU62Z5QQOLXLB6JKTXO33PKU4ANMKUZHEFONKLAFMDSK3DAQ, seg_pos={0 0}, upload=273826]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 5.161.89.252:36920->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:18.735560308 (263h18m59.466827959s) [seg_pos={0 0}, upload=273826]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:18.796277230 (263h18m59.527544891s) [upload=273826]: Done uploading segments... Aug 19 16:45:18.796312307 (263h18m59.527579957s) [upload=273826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 367.191832ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)