Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....X................................] duration: 425.292135ms FAIL (2025-08-19 17:10:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 425.311662ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 335.97781ms FAIL (2025-08-19 17:10:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 335.998309ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X............................................X.] duration: 238.525284ms FAIL (2025-08-19 17:10:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 238.549299ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 845.955753ms FAIL (2025-08-19 17:10:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 845.986351ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration: 364.652622ms FAIL (2025-08-19 17:10:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 364.696075ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 504.216977ms FAIL (2025-08-19 17:10:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.255188ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 498.271784ms FAIL (2025-08-19 17:09:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 498.293384ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X............................................X] duration: 83.316574ms FAIL (2025-08-19 17:09:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:09:30.253040626 (263h43m10.984308276s) [upload=274259]: Starting upload Aug 19 17:09:30.253100950 (263h43m10.9843686s) [upload=274259]: Done starting upload Aug 19 17:09:30.253173446 (263h43m10.984441096s) [upload=274259]: Uploading segments... Aug 19 17:09:30.253221085 (263h43m10.984488735s) [upload=274259]: Got next segment. Inline: true Aug 19 17:09:30.253226325 (263h43m10.984493976s) [upload=274259]: Waiting for error group managing segments... Aug 19 17:09:30.253267713 (263h43m10.984535363s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:09:30.253272942 (263h43m10.984540593s) [upload=274259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:09:30.253276609 (263h43m10.984544259s) [upload=274259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:09:30.321353087 (263h43m11.052620737s) [upload=274259]: Done uploading segments... Aug 19 17:09:30.321371873 (263h43m11.052639532s) [upload=274259]: Upload finished. err: metaclient: Too Many Requests Duration:* 83.338456ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 393.762291ms FAIL (2025-08-19 17:09:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 393.782619ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 575.910512ms FAIL (2025-08-19 17:09:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 575.958151ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 372.192287ms FAIL (2025-08-19 17:09:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 372.23095ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 454.309349ms FAIL (2025-08-19 17:08:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 454.343875ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.976420837s FAIL (2025-08-19 17:08:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 1.97645417s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 196.216846ms FAIL (2025-08-19 17:08:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 196.237716ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 402.709676ms FAIL (2025-08-19 17:08:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 402.729774ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 544.829928ms FAIL (2025-08-19 17:08:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 544.853742ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....X............................] duration: 349.940628ms FAIL (2025-08-19 17:08:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 349.965617ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.721406271s FAIL (2025-08-19 17:07:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 1.721432309s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 456.346437ms FAIL (2025-08-19 17:07:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 456.378717ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 713.40069ms FAIL (2025-08-19 17:07:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 713.427772ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration: 384.833283ms FAIL (2025-08-19 17:07:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 384.860866ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 456.609404ms FAIL (2025-08-19 17:07:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.657123ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 272.884685ms FAIL (2025-08-19 17:06:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 272.899483ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 383.27141ms FAIL (2025-08-19 17:06:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 383.288543ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 612.792548ms FAIL (2025-08-19 17:06:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 612.818015ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 479.969201ms FAIL (2025-08-19 17:06:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.991393ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 337.85802ms FAIL (2025-08-19 17:06:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 337.91648ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 380.761531ms FAIL (2025-08-19 17:06:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 380.789053ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 920.176308ms FAIL (2025-08-19 17:05:58 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 920.199059ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 328.41447ms FAIL (2025-08-19 17:05:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 328.456088ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 468.211312ms FAIL (2025-08-19 17:05:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 468.232301ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 469.257708ms FAIL (2025-08-19 17:05:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 469.276412ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 414.522689ms FAIL (2025-08-19 17:05:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 414.53957ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 192.622354ms FAIL (2025-08-19 17:05:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 192.650216ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....X........................] duration: 479.337616ms FAIL (2025-08-19 17:04:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 479.362393ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 528.157928ms FAIL (2025-08-19 17:04:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 528.176974ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 548.970107ms FAIL (2025-08-19 17:04:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 548.987251ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 156.808537ms FAIL (2025-08-19 17:04:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 156.837943ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration: 429.469352ms FAIL (2025-08-19 17:03:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 429.529786ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 732.533502ms FAIL (2025-08-19 17:03:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 732.582936ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 587.429826ms FAIL (2025-08-19 17:03:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 587.461144ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 346.241669ms FAIL (2025-08-19 17:03:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 346.264473ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 337.576461ms FAIL (2025-08-19 17:03:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 337.59148ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X....X......................] duration: 392.623425ms FAIL (2025-08-19 17:02:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 392.637161ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 583.121068ms FAIL (2025-08-19 17:02:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 583.148459ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 223.754911ms FAIL (2025-08-19 17:02:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 16:39:16.709088169 (263h12m57.440355799s) [upload=273733]: Starting upload Aug 19 16:39:16.709120359 (263h12m57.440387979s) [upload=273733]: Done starting upload Aug 19 16:39:16.709156888 (263h12m57.440424518s) [upload=273733]: Uploading segments... Aug 19 16:39:16.709189779 (263h12m57.440457399s) [upload=273733]: Got next segment. Inline: true Aug 19 16:39:16.709193487 (263h12m57.440461117s) [upload=273733]: Waiting for error group managing segments... Aug 19 16:39:16.709225406 (263h12m57.440493036s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:39:16.709228953 (263h12m57.440496583s) [upload=273733]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:39:16.709231257 (263h12m57.440498887s) [upload=273733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:39:17.168413966 (263h12m57.899681626s) [upload=273733]: Done uploading segments... Aug 19 16:39:17.168439514 (263h12m57.899707164s) [upload=273733]: Upload finished. err: <nil> Duration:* 223.779407ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X....X.....................] duration: 397.173329ms FAIL (2025-08-19 17:02:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 397.18995ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....X....................] duration: 331.428942ms FAIL (2025-08-19 17:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440439150 (263h22m29.171706769s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440660145 (263h22m29.171927765s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440690993 (263h22m29.171958613s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.440892572 (263h22m29.172160192s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441093340 (263h22m29.17236097s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441169954 (263h22m29.172437574s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441405507 (263h22m29.172673137s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441449650 (263h22m29.17271728s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441634066 (263h22m29.172901696s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441871643 (263h22m29.173139263s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.441878365 (263h22m29.173145995s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.442236228 (263h22m29.173503848s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.444182237 (263h22m29.175449867s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.445002428 (263h22m29.176270058s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Uploading piece... Aug 19 16:48:48.521537208 (263h22m29.252804838s) [upload=273893]: Next returned nil segment Aug 19 16:48:48.521548239 (263h22m29.252815869s) [upload=273893]: Waiting for error group managing segments... Aug 19 16:48:48.541405319 (263h22m29.272672938s) [noise=true , address=104.36.82.4:4801 , piece_id=U7MZUQQMFM62TRK2FKUM4EXWCFPF37HPRXKUDSNJRGIFD2I7CRKA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.545658234 (263h22m29.276925864s) [noise=true , address=104.36.82.4:4803 , piece_id=WZUVKIJLQQ7KHHWTJOCANFAQUR6K6SZIV5D4ERBI5C6XWWXOB4SA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.557936526 (263h22m29.289204156s) [noise=true , address=104.36.82.3:4765 , piece_id=FICBIS72PKYKS6AOUULD7HW3YHBWKFD47XDRNWXQ2SUCHEAPCFXQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:48:48.564988173 (263h22m29.296255792s) [noise=true , address=104.36.82.13:4186 , piece_id=ZESX7QFJI6TEZW4WLDEOR6FPWA4NFY2MMSCHBMY7TATXRZRUFNDA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: <nil> Aug 19 16:49:18.351552468 (263h22m59.082820118s) [noise=true , address=45.91.116.4:4938 , piece_id=ZNNQF36YF5PYBM4UVIBXW3J3JDQA5KSOR4QX7IFZFJ44PFO5A47A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351637016 (263h22m59.082904667s) [noise=true , address=77.90.31.15:4697 , piece_id=JD3LN3TJRZBVHDMCTRQDFLJUKWLZ2IMCC3YO7QNAXDRKE5F53OYA, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351690206 (263h22m59.082957866s) [noise=true , address=77.90.31.19:4842 , piece_id=JKRQJGZHUB2ATULVRJUG2X4NKN4FEZX76ZX6UMSKQJCMG44M6GQQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351763084 (263h22m59.083030734s) [noise=true , address=45.91.116.7:4009 , piece_id=4B2VSIA2WUWM63H2ECCN2LPKK4Y4P4QJAZSKU3N3OHS5VAQ4B64Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351882037 (263h22m59.083149697s) [noise=true , address=45.91.116.6:4975 , piece_id=DJQ6P2UCKB3LQSB7AXSD4OPYJZBRKQLT4Q4MDKMRBEFI7WBBJQ7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 5.161.89.252:48890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.351982256 (263h22m59.083249916s) [noise=true , address=45.91.116.4:4939 , piece_id=ZM5HC2WHMWXI5PKNIBWY7CQFTTLKYGDS5JTCDW7VXVFHSUH5VFOQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: rpc: dial tcp 45.91.116.4:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352020067 (263h22m59.083287717s) [noise=true , address=45.91.116.11:4117 , piece_id=KMZCIAUNTAPDMVJPET664DJPPOPT246HG7CPGLFSMSU6PYJPBG7Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352157445 (263h22m59.083425104s) [noise=true , address=45.91.116.13:4153 , piece_id=3U2Q2V4STGSAQMCBABLU2MTFFJHE4SVM6KDPIB65BCJO2YKHHQVQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 5.161.89.252:14982->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352213380 (263h22m59.083481031s) [noise=true , address=45.91.116.10:4083 , piece_id=DCGONYE5WFQMJY5OIJA2GQRWCE5DWRYHASKZL6MQ3E7D3NN4FW4Q, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 5.161.89.252:12880->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352259367 (263h22m59.083527016s) [noise=true , address=77.90.31.17:4770 , piece_id=Y6E4P2UQ2UWLUEXFZBTYOP5FEU3BBVOU4ROP7BE23WWJCGWGGXPQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:20282->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352279484 (263h22m59.083547134s) [noise=true , address=77.90.31.13:4627 , piece_id=C76Y3OG643BPEB4KDBB77CSXZOYKKZLUJT2UUNZBRM4SSJBAXC2A, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 5.161.89.252:29268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352328276 (263h22m59.083595926s) [noise=true , address=77.90.31.14:4661 , piece_id=4HJKLDK4ATJU4IWPD2F45TCJSYXNLO2YCC2RTMK3CGARSRJKSBWQ, seg_pos={0 0}, upload=273893]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:26624->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:18.352362249 (263h22m59.0836299s) [seg_pos={0 0}, upload=273893]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 19 16:49:18.364568045 (263h22m59.095835696s) [upload=273893]: Done uploading segments... Aug 19 16:49:18.364591099 (263h22m59.095858749s) [upload=273893]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 331.473476ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 1.159676613s FAIL (2025-08-19 17:02:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.159694727s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 951.507787ms FAIL (2025-08-19 17:01:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 16:54:39.768694265 (263h28m20.499961914s) [upload=273997]: Starting upload Aug 19 16:54:39.768758044 (263h28m20.500025694s) [upload=273997]: Done starting upload Aug 19 16:54:39.768833737 (263h28m20.500101387s) [upload=273997]: Uploading segments... Aug 19 16:54:39.769226274 (263h28m20.500493924s) [upload=273997]: Got next segment. Inline: false Aug 19 16:54:39.769329438 (263h28m20.500597089s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:54:39.769336001 (263h28m20.500603651s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:54:39.892373182 (263h28m20.623640832s) [seg_pos={0 0}, upload=273997]: Begin upload segment... Aug 19 16:54:39.892431482 (263h28m20.623699132s) [seg_pos={0 0}, upload=273997]: Done begin upload segment. Aug 19 16:54:39.892461147 (263h28m20.623728787s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892496664 (263h28m20.623764314s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.892931502 (263h28m20.624199151s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893245521 (263h28m20.624513162s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893286448 (263h28m20.624554098s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893651074 (263h28m20.624918724s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.893836894 (263h28m20.625104543s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894009988 (263h28m20.625277639s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894377059 (263h28m20.625644699s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894431541 (263h28m20.625699191s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.894732286 (263h28m20.625999926s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895057227 (263h28m20.626324877s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895256040 (263h28m20.626523701s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.895379082 (263h28m20.626646722s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896068288 (263h28m20.627335948s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.896754376 (263h28m20.628022026s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897242674 (263h28m20.628510314s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.897649399 (263h28m20.628917049s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Uploading piece... Aug 19 16:54:39.996733958 (263h28m20.728001578s) [upload=273997]: Next returned nil segment Aug 19 16:54:39.996744999 (263h28m20.728012619s) [upload=273997]: Waiting for error group managing segments... Aug 19 16:54:40.006905447 (263h28m20.738173067s) [noise=true , address=45.91.116.14:4206 , piece_id=AI3EDHQQH6TK7EELZMDQ3CH7FSFS5BWMEA3O2RUNF7OPVSGBDCWA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.010650045 (263h28m20.741917695s) [noise=true , address=104.36.82.49:4002 , piece_id=BON56TN53YTZ42H3UGSCYPBXH7T3YYIHLVNURECRGVGCSH2XF6CQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012387941 (263h28m20.743655582s) [noise=true , address=104.36.82.11:4128 , piece_id=GOLKDJAL3T5V3KM4UDLE7PO3GH4SJCWUZMBTG4QE6LAXDR3KLPDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.012600531 (263h28m20.743868191s) [noise=true , address=45.91.116.11:4145 , piece_id=HIPVV7SVTNCJ3RTPT7GZOL7FKY4BYEOIBACHGU426RTAG7GXH6MA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.013127341 (263h28m20.744394981s) [noise=true , address=45.91.116.11:4128 , piece_id=YLO2NLMW7S2RETXLR4KCBTWI4RHY6I4CYKCDXV7MT2Z2TXUZECKQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.015725054 (263h28m20.746992704s) [noise=true , address=79.127.219.51:4200 , piece_id=OWKU7IP27IDNW7CIFG6CBPVDZRBR72OCQ7F6XX6EMUWWEEM5KTFA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.017748017 (263h28m20.749015658s) [noise=true , address=45.91.116.10:4084 , piece_id=L5STTBHOA4FX5GPTHWWPC5RWWSVGIJE7VWDN5WRCDP6IS2GBPCJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.022367990 (263h28m20.75363564s) [noise=true , address=79.127.219.47:4048 , piece_id=QTLYJYLL4D5PDV46IQBVUVMNOXKPGQCNRZ2SGJPWRL5PE5WDEXLA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023459282 (263h28m20.754726912s) [noise=true , address=104.36.82.60:4357 , piece_id=DHMHWCJC4FXOXN2MJ2KZE5M3NEAUGZBM7UIOWLOB77UFWBWEIHCA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.023905380 (263h28m20.755173031s) [noise=true , address=77.90.31.12:4619 , piece_id=G5CSYE7LDKSN2NDWGGLG232TC6HIFMD5ZIJOH5IJBPP5P4RHTVDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.028678080 (263h28m20.759945721s) [noise=true , address=104.36.82.34:4562 , piece_id=A4ZF6PYWLA3R3TSNWSEMWXEYDCJJXYKEXZ7XCTR32G34YLLBPVUA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.035459938 (263h28m20.766727568s) [noise=true , address=77.90.31.17:4769 , piece_id=DVRHDX7IG4BLA3YABZAMINADGDYYQHCL5PG7RHQY5K5WC2RQ7BPQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.038353317 (263h28m20.769620966s) [noise=true , address=77.90.31.19:4860 , piece_id=H4QRMESKHG2VBNW6IPZRTTXMKCG5KT2EUBB3XW4SNQL5IWV7V4XQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045552748 (263h28m20.776820388s) [noise=true , address=104.36.82.36:4627 , piece_id=TZGWDULYRO6X7IOIMD5NXVKA67MBD3PUXODS5UDCYIE2FFDP4DMQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: <nil> Aug 19 16:54:40.045575121 (263h28m20.776842761s) [seg_pos={0 0}, upload=273997]: Segment reached optimal threshold of 14 pieces. Aug 19 16:54:40.045725383 (263h28m20.776993023s) [noise=true , address=104.36.82.23:4458 , piece_id=JRGAVETEMJJF3DNXAUDGVNCKZBW574VHBA3HMSQERYFBTXFVRAJQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045862330 (263h28m20.777129961s) [noise=true , address=77.90.31.11:4579 , piece_id=E72566DABDPJRFHOB2BUUUMRM4I23RO2DLIMYPRILMAJDBX54XDA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.045955475 (263h28m20.777223116s) [noise=true , address=79.127.201.215:4352 , piece_id=DXLVEBP2YYIUME2KXIUHOCH33FH6P4MEJD7EOP6BCNEJLJ37Y3RA, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7): protocol: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:60156->79.127.201.215:4352: use of closed network connection Aug 19 16:54:40.045990340 (263h28m20.777257991s) [noise=true , address=77.90.31.19:4858 , piece_id=H2FLBEJW5FVFFY7Y7MI3KTQR36W5NAAVWZ3XJ4DQFP27DPH2DSVQ, seg_pos={0 0}, upload=273997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 19 16:54:40.046041047 (263h28m20.777308707s) [seg_pos={0 0}, upload=273997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:54:40.046213360 (263h28m20.77748101s) [upload=273997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:54:40.046222577 (263h28m20.777490227s) [upload=273997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:54:40.507200222 (263h28m21.238467872s) [upload=273997]: Done uploading segments... Aug 19 16:54:40.507219628 (263h28m21.238487279s) [upload=273997]: Upload finished. err: <nil> Duration: 951.53029ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)