Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 1m0.000778877s FAIL (2025-08-07 21:21:30 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ading segments... Aug 7 21:11:31.682853265 (4h44m45.269909476s) [upload=4908]: Upload finished. err: <nil> Aug 7 21:11:51.754258397 (4h45m5.341314618s) [upload=4915]: Starting upload Aug 7 21:11:51.754330331 (4h45m5.341386542s) [upload=4915]: Done starting upload Aug 7 21:11:51.754383972 (4h45m5.341440193s) [upload=4915]: Uploading segments... Aug 7 21:11:51.754417083 (4h45m5.341473294s) [upload=4915]: Got next segment. Inline: true Aug 7 21:11:51.754420630 (4h45m5.34147684s) [upload=4915]: Waiting for error group managing segments... Aug 7 21:11:51.754453131 (4h45m5.341509352s) [upload=4915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:51.754457108 (4h45m5.341513329s) [upload=4915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:51.754460725 (4h45m5.341516936s) [upload=4915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:51.830658438 (4h45m5.417714649s) [upload=4915]: Done uploading segments... Aug 7 21:11:51.830673094 (4h45m5.417729315s) [upload=4915]: Upload finished. err: <nil> Aug 7 21:12:54.778494646 (4h46m8.365550856s) [upload=4932]: Starting upload Aug 7 21:12:54.778560719 (4h46m8.36561693s) [upload=4932]: Done starting upload Aug 7 21:12:54.778608638 (4h46m8.365664859s) [upload=4932]: Uploading segments... Aug 7 21:12:54.778637322 (4h46m8.365693543s) [upload=4932]: Got next segment. Inline: true Aug 7 21:12:54.778640999 (4h46m8.36569722s) [upload=4932]: Waiting for error group managing segments... Aug 7 21:12:54.778677527 (4h46m8.365733739s) [upload=4932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:54.778681655 (4h46m8.365737876s) [upload=4932]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:54.778684771 (4h46m8.365740983s) [upload=4932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.868239378 (4h46m8.455295599s) [upload=4932]: Done uploading segments... Aug 7 21:12:54.868253865 (4h46m8.455310086s) [upload=4932]: Upload finished. err: <nil> Aug 7 21:13:23.238906839 (4h46m36.82596306s) [upload=4942]: Starting upload Aug 7 21:13:23.238959356 (4h46m36.826015567s) [upload=4942]: Done starting upload Aug 7 21:13:23.239011013 (4h46m36.826067234s) [upload=4942]: Uploading segments... Aug 7 21:13:23.239039727 (4h46m36.826095937s) [upload=4942]: Got next segment. Inline: true Aug 7 21:13:23.239043483 (4h46m36.826099695s) [upload=4942]: Waiting for error group managing segments... Aug 7 21:13:23.239098526 (4h46m36.826154737s) [upload=4942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:23.239104707 (4h46m36.826160919s) [upload=4942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:23.239108474 (4h46m36.826164696s) [upload=4942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:23.309357825 (4h46m36.896414127s) [upload=4942]: Done uploading segments... Aug 7 21:13:23.309383974 (4h46m36.896440216s) [upload=4942]: Upload finished. err: <nil> Aug 7 21:14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:30.042411605 (4h52m43.629467826s) [upload=5040]: Starting upload Aug 7 21:19:30.042491373 (4h52m43.629547594s) [upload=5040]: Done starting upload Aug 7 21:19:30.042544984 (4h52m43.629601195s) [upload=5040]: Uploading segments... Aug 7 21:19:30.042585870 (4h52m43.629642091s) [upload=5040]: Got next segment. Inline: true Aug 7 21:19:30.042590809 (4h52m43.62964703s) [upload=5040]: Waiting for error group managing segments... Aug 7 21:19:30.042637226 (4h52m43.629693436s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:30.042642325 (4h52m43.629698546s) [upload=5040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:30.042647144 (4h52m43.629703365s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:30.020861737 (4h53m43.607917958s) [upload=5040]: Done uploading segments... Aug 7 21:20:30.020878959 (4h53m43.60793518s) [upload=5040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:30.129088028 (4h53m43.71614424s) [upload=5055]: Starting upload Aug 7 21:20:30.129144182 (4h53m43.716200404s) [upload=5055]: Done starting upload Aug 7 21:20:30.129191110 (4h53m43.716247332s) [upload=5055]: Uploading segments... Aug 7 21:20:30.129220234 (4h53m43.716276455s) [upload=5055]: Got next segment. Inline: true Aug 7 21:20:30.129223841 (4h53m43.716280062s) [upload=5055]: Waiting for error group managing segments... Aug 7 21:20:30.129266070 (4h53m43.716322282s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.129269998 (4h53m43.716326219s) [upload=5055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:30.129272523 (4h53m43.716328734s) [upload=5055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:30.117153252 (4h54m43.704209473s) [upload=5055]: Done uploading segments... Aug 7 21:21:30.117172058 (4h54m43.704228279s) [upload=5055]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000791231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 30.001390743s FAIL (2025-08-07 21:21:24 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ]: Upload finished. err: context deadline exceeded Aug 7 21:18:36.577784924 (4h51m50.164841166s) [upload=5025]: Starting upload Aug 7 21:18:36.577862118 (4h51m50.164918349s) [upload=5025]: Done starting upload Aug 7 21:18:36.577953218 (4h51m50.165009459s) [upload=5025]: Uploading segments... Aug 7 21:18:36.578218493 (4h51m50.165274724s) [upload=5025]: Got next segment. Inline: false Aug 7 21:18:36.578268738 (4h51m50.165324978s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:36.578284077 (4h51m50.165340317s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:54.119485924 (4h52m7.706542165s) [seg_pos={0 0}, upload=5025]: Begin upload segment... Aug 7 21:18:54.119532450 (4h52m7.706588691s) [seg_pos={0 0}, upload=5025]: Done begin upload segment. Aug 7 21:18:54.119561394 (4h52m7.706617645s) [noise=true , address=45.91.116.7:4011 , piece_id=H2JRFAYC7KLC3TIXQ4PJZTKZLY552JJMZF6XTCFMHFD3RGIUMOCQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.119574769 (4h52m7.70663101s) [noise=true , address=45.91.116.8:4047 , piece_id=7F4MB4X3CJZH3DUXC7VXN3HGQBBRFUV62G4KLCKV7BSARABJENWQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120057159 (4h52m7.7071134s) [noise=true , address=104.36.82.4:4803 , piece_id=DCMHD25G5K5S33FQ72K7VEQ3U6ZV5UBZK5UFVDWUI5I3BX6E4NHA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120241273 (4h52m7.707297534s) [noise=true , address=104.36.82.6:4876 , piece_id=6WRCDS47O3FUJLE2BT6QV3OOXKGONXLHZ7TMPXKHH4LP2AJGHV5Q, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120508351 (4h52m7.707564592s) [noise=true , address=104.36.82.15:4259 , piece_id=ZNJ3QCHTB2LJVM7KI52E3RPGDYEHWWFTEEBBNUIOVPIDOQP7VTKQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120776011 (4h52m7.707832252s) [noise=true , address=77.90.31.15:4697 , piece_id=PILNZYYWJABT57ITSG77YXQZN7EJKSDHB75RJHE6SPB67FSWCPGQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120880315 (4h52m7.707936557s) [noise=true , address=45.91.116.14:4189 , piece_id=GRDSWIHHZ4FEWBEHRY7WPEF546PJWGBIXXCYDESVV4N3GSAWVGYA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121253463 (4h52m7.708309713s) [noise=true , address=45.91.116.10:4083 , piece_id=HW5YZYWXC7XHQA37JIKXDA7SSO6HOVCCGDS2SHWP7ROPG6MHQ7ZA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121362616 (4h52m7.708418858s) [noise=true , address=77.90.31.10:4520 , piece_id=Z5P2H3YFJDC5WMKMFIFVYGFVAU2R27KTNMIAYBDYZGRUT5TWN5QQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121608686 (4h52m7.708664927s) [noise=true , address=77.90.31.11:4555 , piece_id=D7YHBQ32VTIJEC4HKWQYHWMOULL3KNN7P7TJSA7LV4TLUICYGWBQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121894870 (4h52m7.708951121s) [noise=true , address=45.91.116.8:4048 , piece_id=2IZI3WOTOBBKYHKVPJDQ6PGS5VTDAFIGTI373VWKT7QLAZR7W3FA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121987422 (4h52m7.709043694s) [noise=true , address=77.90.31.19:4841 , piece_id=AUK7XFVC5R73KNHLVVJVRGUTRWBEJ6LP4FZIUJ77DECJRXMHWRRA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.122419388 (4h52m7.70947563s) [noise=true , address=104.36.82.15:4258 , piece_id=VT3VKJFUHTSDCNR7PIBIJXEDYG3HGIVT4JZRR5VM3WJ6A5RHQQZQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.122822722 (4h52m7.709878973s) [noise=true , address=77.90.31.13:4628 , piece_id=N3UNNLVQJ3GALE77VUERGE4PJDIYJK4T5VJEARML2QSMNJ2AAPJA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.123209864 (4h52m7.710266105s) [noise=true , address=45.91.116.5:4297 , piece_id=YK7QNB23S43MPEVNDLAT5WBFAMVBPQ3CVYKEMW5VBUHUAI5W4S5Q, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.123571199 (4h52m7.710627439s) [noise=true , address=104.36.82.14:4223 , piece_id=LX33NZFO2TFV43ARCAIGFYXJZZRR633JWZTHSO6XWSRP7TQXU5HQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.259677990 (4h52m7.846734232s) [upload=5025]: Next returned nil segment Aug 7 21:18:54.259835595 (4h52m7.846891847s) [upload=5025]: Waiting for error group managing segments... Aug 7 21:18:54.269953672 (4h52m7.857009913s) [noise=true , address=45.91.116.7:4011 , piece_id=H2JRFAYC7KLC3TIXQ4PJZTKZLY552JJMZF6XTCFMHFD3RGIUMOCQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.270112598 (4h52m7.857168849s) [noise=true , address=45.91.116.8:4048 , piece_id=2IZI3WOTOBBKYHKVPJDQ6PGS5VTDAFIGTI373VWKT7QLAZR7W3FA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.270947997 (4h52m7.858004239s) [noise=true , address=45.91.116.5:4297 , piece_id=YK7QNB23S43MPEVNDLAT5WBFAMVBPQ3CVYKEMW5VBUHUAI5W4S5Q, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.271046852 (4h52m7.858103094s) [noise=true , address=77.90.31.13:4628 , piece_id=N3UNNLVQJ3GALE77VUERGE4PJDIYJK4T5VJEARML2QSMNJ2AAPJA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.272223499 (4h52m7.85927975s) [noise=true , address=45.91.116.10:4083 , piece_id=HW5YZYWXC7XHQA37JIKXDA7SSO6HOVCCGDS2SHWP7ROPG6MHQ7ZA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.272541713 (4h52m7.859597954s) [noise=true , address=45.91.116.14:4189 , piece_id=GRDSWIHHZ4FEWBEHRY7WPEF546PJWGBIXXCYDESVV4N3GSAWVGYA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.273006260 (4h52m7.860062501s) [noise=true , address=45.91.116.8:4047 , piece_id=7F4MB4X3CJZH3DUXC7VXN3HGQBBRFUV62G4KLCKV7BSARABJENWQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.275575946 (4h52m7.862632188s) [noise=true , address=77.90.31.10:4520 , piece_id=Z5P2H3YFJDC5WMKMFIFVYGFVAU2R27KTNMIAYBDYZGRUT5TWN5QQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.281367938 (4h52m7.868424169s) [noise=true , address=77.90.31.15:4697 , piece_id=PILNZYYWJABT57ITSG77YXQZN7EJKSDHB75RJHE6SPB67FSWCPGQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.284511737 (4h52m7.871567998s) [noise=true , address=77.90.31.19:4841 , piece_id=AUK7XFVC5R73KNHLVVJVRGUTRWBEJ6LP4FZIUJ77DECJRXMHWRRA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.291770877 (4h52m7.878827127s) [noise=true , address=104.36.82.4:4803 , piece_id=DCMHD25G5K5S33FQ72K7VEQ3U6ZV5UBZK5UFVDWUI5I3BX6E4NHA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.298214294 (4h52m7.885270545s) [noise=true , address=77.90.31.11:4555 , piece_id=D7YHBQ32VTIJEC4HKWQYHWMOULL3KNN7P7TJSA7LV4TLUICYGWBQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532849875 (4h52m8.119906117s) [noise=true , address=104.36.82.6:4876 , piece_id=6WRCDS47O3FUJLE2BT6QV3OOXKGONXLHZ7TMPXKHH4LP2AJGHV5Q, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532867749 (4h52m8.11992399s) [seg_pos={0 0}, upload=5025]: Segment reached optimal threshold of 13 pieces. Aug 7 21:18:54.533278306 (4h52m8.120334548s) [noise=true , address=104.36.82.15:4259 , piece_id=ZNJ3QCHTB2LJVM7KI52E3RPGDYEHWWFTEEBBNUIOVPIDOQP7VTKQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533410502 (4h52m8.120466743s) [noise=true , address=104.36.82.15:4258 , piece_id=VT3VKJFUHTSDCNR7PIBIJXEDYG3HGIVT4JZRR5VM3WJ6A5RHQQZQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533558579 (4h52m8.12061482s) [noise=true , address=104.36.82.14:4223 , piece_id=LX33NZFO2TFV43ARCAIGFYXJZZRR633JWZTHSO6XWSRP7TQXU5HQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection Aug 7 21:18:54.533575210 (4h52m8.120631451s) [seg_pos={0 0}, upload=5025]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:18:54.533666891 (4h52m8.120723132s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:54.533672582 (4h52m8.120728823s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:54.197308723 (4h54m7.784364965s) [upload=5025]: Done uploading segments... Aug 7 21:20:54.197331416 (4h54m7.784387657s) [upload=5025]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:54.289617342 (4h54m7.876673583s) [upload=5062]: Starting upload Aug 7 21:20:54.289682233 (4h54m7.876738475s) [upload=5062]: Done starting upload Aug 7 21:20:54.290100865 (4h54m7.877157115s) [upload=5062]: Uploading segments... Aug 7 21:20:54.290145758 (4h54m7.877201999s) [upload=5062]: Got next segment. Inline: false Aug 7 21:20:54.290178920 (4h54m7.877235161s) [upload=5062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:54.290184501 (4h54m7.877240741s) [upload=5062]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:24.274797304 (4h54m37.861853546s) [upload=5062]: Next segment err: context deadline exceeded Aug 7 21:21:24.274996005 (4h54m37.862052246s) [upload=5062]: Done uploading segments... Aug 7 21:21:24.275019569 (4h54m37.86207581s) [upload=5062]: Upload finished. err: context deadline exceeded Duration: 30.001428594s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.000889999s FAIL (2025-08-07 21:21:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000906059s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.148851663s FAIL (2025-08-07 21:21:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 1.14887671s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.00150349s FAIL (2025-08-07 21:21:08 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001524901s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 30.000888833s FAIL (2025-08-07 21:21:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000900013s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..............................................XX..XXXXXXXXXX] duration: 30.000810992s FAIL (2025-08-07 21:20:58 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000851998s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 2m17.63331805s FAIL (2025-08-07 21:20:54 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad finished. err: context deadline exceeded Aug 7 21:17:29.086779152 (4h50m42.673835373s) [upload=5004]: Starting upload Aug 7 21:17:29.086833233 (4h50m42.673889444s) [upload=5004]: Done starting upload Aug 7 21:17:29.086916819 (4h50m42.67397304s) [upload=5004]: Uploading segments... Aug 7 21:17:29.087068823 (4h50m42.674125034s) [upload=5004]: Got next segment. Inline: false Aug 7 21:17:29.087119998 (4h50m42.674176219s) [upload=5004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.087124536 (4h50m42.674180748s) [upload=5004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:59.072413634 (4h51m12.659469875s) [upload=5004]: Next segment err: context deadline exceeded Aug 7 21:17:59.072514252 (4h51m12.659570503s) [upload=5004]: Done uploading segments... Aug 7 21:17:59.072534319 (4h51m12.659590571s) [upload=5004]: Upload finished. err: context deadline exceeded Aug 7 21:18:36.577784924 (4h51m50.164841166s) [upload=5025]: Starting upload Aug 7 21:18:36.577862118 (4h51m50.164918349s) [upload=5025]: Done starting upload Aug 7 21:18:36.577953218 (4h51m50.165009459s) [upload=5025]: Uploading segments... Aug 7 21:18:36.578218493 (4h51m50.165274724s) [upload=5025]: Got next segment. Inline: false Aug 7 21:18:36.578268738 (4h51m50.165324978s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:36.578284077 (4h51m50.165340317s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:54.119485924 (4h52m7.706542165s) [seg_pos={0 0}, upload=5025]: Begin upload segment... Aug 7 21:18:54.119532450 (4h52m7.706588691s) [seg_pos={0 0}, upload=5025]: Done begin upload segment. Aug 7 21:18:54.119561394 (4h52m7.706617645s) [noise=true , address=45.91.116.7:4011 , piece_id=H2JRFAYC7KLC3TIXQ4PJZTKZLY552JJMZF6XTCFMHFD3RGIUMOCQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.119574769 (4h52m7.70663101s) [noise=true , address=45.91.116.8:4047 , piece_id=7F4MB4X3CJZH3DUXC7VXN3HGQBBRFUV62G4KLCKV7BSARABJENWQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120057159 (4h52m7.7071134s) [noise=true , address=104.36.82.4:4803 , piece_id=DCMHD25G5K5S33FQ72K7VEQ3U6ZV5UBZK5UFVDWUI5I3BX6E4NHA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120241273 (4h52m7.707297534s) [noise=true , address=104.36.82.6:4876 , piece_id=6WRCDS47O3FUJLE2BT6QV3OOXKGONXLHZ7TMPXKHH4LP2AJGHV5Q, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120508351 (4h52m7.707564592s) [noise=true , address=104.36.82.15:4259 , piece_id=ZNJ3QCHTB2LJVM7KI52E3RPGDYEHWWFTEEBBNUIOVPIDOQP7VTKQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120776011 (4h52m7.707832252s) [noise=true , address=77.90.31.15:4697 , piece_id=PILNZYYWJABT57ITSG77YXQZN7EJKSDHB75RJHE6SPB67FSWCPGQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.120880315 (4h52m7.707936557s) [noise=true , address=45.91.116.14:4189 , piece_id=GRDSWIHHZ4FEWBEHRY7WPEF546PJWGBIXXCYDESVV4N3GSAWVGYA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121253463 (4h52m7.708309713s) [noise=true , address=45.91.116.10:4083 , piece_id=HW5YZYWXC7XHQA37JIKXDA7SSO6HOVCCGDS2SHWP7ROPG6MHQ7ZA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121362616 (4h52m7.708418858s) [noise=true , address=77.90.31.10:4520 , piece_id=Z5P2H3YFJDC5WMKMFIFVYGFVAU2R27KTNMIAYBDYZGRUT5TWN5QQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121608686 (4h52m7.708664927s) [noise=true , address=77.90.31.11:4555 , piece_id=D7YHBQ32VTIJEC4HKWQYHWMOULL3KNN7P7TJSA7LV4TLUICYGWBQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121894870 (4h52m7.708951121s) [noise=true , address=45.91.116.8:4048 , piece_id=2IZI3WOTOBBKYHKVPJDQ6PGS5VTDAFIGTI373VWKT7QLAZR7W3FA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.121987422 (4h52m7.709043694s) [noise=true , address=77.90.31.19:4841 , piece_id=AUK7XFVC5R73KNHLVVJVRGUTRWBEJ6LP4FZIUJ77DECJRXMHWRRA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.122419388 (4h52m7.70947563s) [noise=true , address=104.36.82.15:4258 , piece_id=VT3VKJFUHTSDCNR7PIBIJXEDYG3HGIVT4JZRR5VM3WJ6A5RHQQZQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.122822722 (4h52m7.709878973s) [noise=true , address=77.90.31.13:4628 , piece_id=N3UNNLVQJ3GALE77VUERGE4PJDIYJK4T5VJEARML2QSMNJ2AAPJA, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.123209864 (4h52m7.710266105s) [noise=true , address=45.91.116.5:4297 , piece_id=YK7QNB23S43MPEVNDLAT5WBFAMVBPQ3CVYKEMW5VBUHUAI5W4S5Q, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.123571199 (4h52m7.710627439s) [noise=true , address=104.36.82.14:4223 , piece_id=LX33NZFO2TFV43ARCAIGFYXJZZRR633JWZTHSO6XWSRP7TQXU5HQ, seg_pos={0 0}, upload=5025]: Uploading piece... Aug 7 21:18:54.259677990 (4h52m7.846734232s) [upload=5025]: Next returned nil segment Aug 7 21:18:54.259835595 (4h52m7.846891847s) [upload=5025]: Waiting for error group managing segments... Aug 7 21:18:54.269953672 (4h52m7.857009913s) [noise=true , address=45.91.116.7:4011 , piece_id=H2JRFAYC7KLC3TIXQ4PJZTKZLY552JJMZF6XTCFMHFD3RGIUMOCQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.270112598 (4h52m7.857168849s) [noise=true , address=45.91.116.8:4048 , piece_id=2IZI3WOTOBBKYHKVPJDQ6PGS5VTDAFIGTI373VWKT7QLAZR7W3FA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.270947997 (4h52m7.858004239s) [noise=true , address=45.91.116.5:4297 , piece_id=YK7QNB23S43MPEVNDLAT5WBFAMVBPQ3CVYKEMW5VBUHUAI5W4S5Q, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.271046852 (4h52m7.858103094s) [noise=true , address=77.90.31.13:4628 , piece_id=N3UNNLVQJ3GALE77VUERGE4PJDIYJK4T5VJEARML2QSMNJ2AAPJA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.272223499 (4h52m7.85927975s) [noise=true , address=45.91.116.10:4083 , piece_id=HW5YZYWXC7XHQA37JIKXDA7SSO6HOVCCGDS2SHWP7ROPG6MHQ7ZA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.272541713 (4h52m7.859597954s) [noise=true , address=45.91.116.14:4189 , piece_id=GRDSWIHHZ4FEWBEHRY7WPEF546PJWGBIXXCYDESVV4N3GSAWVGYA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.273006260 (4h52m7.860062501s) [noise=true , address=45.91.116.8:4047 , piece_id=7F4MB4X3CJZH3DUXC7VXN3HGQBBRFUV62G4KLCKV7BSARABJENWQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.275575946 (4h52m7.862632188s) [noise=true , address=77.90.31.10:4520 , piece_id=Z5P2H3YFJDC5WMKMFIFVYGFVAU2R27KTNMIAYBDYZGRUT5TWN5QQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.281367938 (4h52m7.868424169s) [noise=true , address=77.90.31.15:4697 , piece_id=PILNZYYWJABT57ITSG77YXQZN7EJKSDHB75RJHE6SPB67FSWCPGQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.284511737 (4h52m7.871567998s) [noise=true , address=77.90.31.19:4841 , piece_id=AUK7XFVC5R73KNHLVVJVRGUTRWBEJ6LP4FZIUJ77DECJRXMHWRRA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.291770877 (4h52m7.878827127s) [noise=true , address=104.36.82.4:4803 , piece_id=DCMHD25G5K5S33FQ72K7VEQ3U6ZV5UBZK5UFVDWUI5I3BX6E4NHA, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.298214294 (4h52m7.885270545s) [noise=true , address=77.90.31.11:4555 , piece_id=D7YHBQ32VTIJEC4HKWQYHWMOULL3KNN7P7TJSA7LV4TLUICYGWBQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532849875 (4h52m8.119906117s) [noise=true , address=104.36.82.6:4876 , piece_id=6WRCDS47O3FUJLE2BT6QV3OOXKGONXLHZ7TMPXKHH4LP2AJGHV5Q, seg_pos={0 0}, upload=5025]: Done uploading piece. err: <nil> Aug 7 21:18:54.532867749 (4h52m8.11992399s) [seg_pos={0 0}, upload=5025]: Segment reached optimal threshold of 13 pieces. Aug 7 21:18:54.533278306 (4h52m8.120334548s) [noise=true , address=104.36.82.15:4259 , piece_id=ZNJ3QCHTB2LJVM7KI52E3RPGDYEHWWFTEEBBNUIOVPIDOQP7VTKQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533410502 (4h52m8.120466743s) [noise=true , address=104.36.82.15:4258 , piece_id=VT3VKJFUHTSDCNR7PIBIJXEDYG3HGIVT4JZRR5VM3WJ6A5RHQQZQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:54.533558579 (4h52m8.12061482s) [noise=true , address=104.36.82.14:4223 , piece_id=LX33NZFO2TFV43ARCAIGFYXJZZRR633JWZTHSO6XWSRP7TQXU5HQ, seg_pos={0 0}, upload=5025]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47372->104.36.82.14:4223: use of closed network connection Aug 7 21:18:54.533575210 (4h52m8.120631451s) [seg_pos={0 0}, upload=5025]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:18:54.533666891 (4h52m8.120723132s) [upload=5025]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:54.533672582 (4h52m8.120728823s) [upload=5025]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:54.197308723 (4h54m7.784364965s) [upload=5025]: Done uploading segments... Aug 7 21:20:54.197331416 (4h54m7.784387657s) [upload=5025]: Upload finished. err: metaclient: context deadline exceeded Duration: 2m17.633348968s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 30.001160351s FAIL (2025-08-07 21:20:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001181741s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...................................................XX.XXXXXX] duration: 1m0.001418755s FAIL (2025-08-07 21:20:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001480861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 804.832889ms FAIL (2025-08-07 21:20:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 804.864146ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 30.330116ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:20:45 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 30.351165ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.000836593s FAIL (2025-08-07 21:20:33 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== r: <nil> Aug 7 21:12:30.034507380 (4h45m43.62156362s) [upload=4926]: Starting upload Aug 7 21:12:30.034584302 (4h45m43.621640544s) [upload=4926]: Done starting upload Aug 7 21:12:30.034643523 (4h45m43.621699754s) [upload=4926]: Uploading segments... Aug 7 21:12:30.034681965 (4h45m43.621738206s) [upload=4926]: Got next segment. Inline: true Aug 7 21:12:30.034687645 (4h45m43.621743886s) [upload=4926]: Waiting for error group managing segments... Aug 7 21:12:30.034730766 (4h45m43.621787007s) [upload=4926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:30.034735976 (4h45m43.621792207s) [upload=4926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:30.034739642 (4h45m43.621795883s) [upload=4926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:00.022476261 (4h46m13.609532503s) [upload=4926]: Done uploading segments... Aug 7 21:13:00.022497211 (4h46m13.609553452s) [upload=4926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:15.702096720 (4h46m29.289152941s) [upload=4939]: Starting upload Aug 7 21:13:15.702154518 (4h46m29.289210739s) [upload=4939]: Done starting upload Aug 7 21:13:15.702196716 (4h46m29.289252928s) [upload=4939]: Uploading segments... Aug 7 21:13:15.702224368 (4h46m29.289280589s) [upload=4939]: Got next segment. Inline: true Aug 7 21:13:15.702228446 (4h46m29.289284657s) [upload=4939]: Waiting for error group managing segments... Aug 7 21:13:15.702261458 (4h46m29.289317669s) [upload=4939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:15.702265306 (4h46m29.289321527s) [upload=4939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:15.702267880 (4h46m29.289324101s) [upload=4939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.689162551 (4h46m59.276218792s) [upload=4939]: Done uploading segments... Aug 7 21:13:45.689208046 (4h46m59.276264277s) [upload=4939]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:13.522007492 (4h47m27.109063713s) [upload=4955]: Starting upload Aug 7 21:14:13.522102209 (4h47m27.10915843s) [upload=4955]: Done starting upload Aug 7 21:14:13.522154938 (4h47m27.109211179s) [upload=4955]: Uploading segments... Aug 7 21:14:13.522235488 (4h47m27.109291729s) [upload=4955]: Got next segment. Inline: true Aug 7 21:14:13.522242682 (4h47m27.109298922s) [upload=4955]: Waiting for error group managing segments... Aug 7 21:14:13.522293467 (4h47m27.109349708s) [upload=4955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:13.522299518 (4h47m27.109355759s) [upload=4955]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:13.522303275 (4h47m27.109359516s) [upload=4955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:43.507883903 (4h47m57.094940144s) [upload=4955]: Done uploading segments... Aug 7 21:14:43.507905914 (4h47m57.094962156s) [upload=4955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:54.825501522 (4h48m8.412557734s) [upload=4964]: Starting upload Aug 7 21:14:54.825580730 (4h48m8.412636951s) [upload=4964]: Done starting upload Aug 7 21:14:54.825629411 (4h48m8.412685632s) [upload=4964]: Uploading segments... Aug 7 21:14:54.825660400 (4h48m8.41271661s) [upload=4964]: Got next segment. Inline: true Aug 7 21:14:54.825664547 (4h48m8.412720758s) [upload=4964]: Waiting for error group managing segments... Aug 7 21:14:54.825705873 (4h48m8.412762115s) [upload=4964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:54.825711855 (4h48m8.412768076s) [upload=4964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:54.825715662 (4h48m8.412771883s) [upload=4964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:24.812961328 (4h48m38.400017569s) [upload=4964]: Done uploading segments... Aug 7 21:15:24.812988770 (4h48m38.40004501s) [upload=4964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:40.993035983 (4h48m54.580092194s) [upload=4978]: Starting upload Aug 7 21:15:40.993105403 (4h48m54.580161614s) [upload=4978]: Done starting upload Aug 7 21:15:40.993154163 (4h48m54.580210375s) [upload=4978]: Uploading segments... Aug 7 21:15:40.993189519 (4h48m54.580245741s) [upload=4978]: Got next segment. Inline: true Aug 7 21:15:40.993194319 (4h48m54.58025054s) [upload=4978]: Waiting for error group managing segments... Aug 7 21:15:40.993226930 (4h48m54.580283151s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:40.993231167 (4h48m54.580287379s) [upload=4978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:40.993234264 (4h48m54.580290474s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:10.982081529 (4h49m24.569137771s) [upload=4978]: Done uploading segments... Aug 7 21:16:10.982104473 (4h49m24.569160713s) [upload=4978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:20.227399180 (4h49m33.814455401s) [upload=4990]: Starting upload Aug 7 21:16:20.227450334 (4h49m33.814506545s) [upload=4990]: Done starting upload Aug 7 21:16:20.227491060 (4h49m33.814547272s) [upload=4990]: Uploading segments... Aug 7 21:16:20.227516609 (4h49m33.81457283s) [upload=4990]: Got next segment. Inline: true Aug 7 21:16:20.227520536 (4h49m33.814576757s) [upload=4990]: Waiting for error group managing segments... Aug 7 21:16:20.227552426 (4h49m33.814608636s) [upload=4990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:20.227556263 (4h49m33.814612484s) [upload=4990]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:20.227558527 (4h49m33.814614739s) [upload=4990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:50.216081431 (4h50m3.803137652s) [upload=4990]: Done uploading segments... Aug 7 21:16:50.216097492 (4h50m3.803153713s) [upload=4990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:57.322533914 (4h50m10.909590156s) [upload=4995]: Starting upload Aug 7 21:16:57.322609355 (4h50m10.909665607s) [upload=4995]: Done starting upload Aug 7 21:16:57.322672883 (4h50m10.909729125s) [upload=4995]: Uploading segments... Aug 7 21:16:57.322711135 (4h50m10.909767376s) [upload=4995]: Got next segment. Inline: true Aug 7 21:16:57.322716685 (4h50m10.909772927s) [upload=4995]: Waiting for error group managing segments... Aug 7 21:16:57.322761279 (4h50m10.90981752s) [upload=4995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:57.322767100 (4h50m10.909823341s) [upload=4995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:57.322770697 (4h50m10.909826938s) [upload=4995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:27.310589172 (4h50m40.897645383s) [upload=4995]: Done uploading segments... Aug 7 21:17:27.310603289 (4h50m40.897659499s) [upload=4995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:38.458031408 (4h50m52.045087639s) [upload=5010]: Starting upload Aug 7 21:17:38.458135221 (4h50m52.045191452s) [upload=5010]: Done starting upload Aug 7 21:17:38.458199552 (4h50m52.045255784s) [upload=5010]: Uploading segments... Aug 7 21:17:38.458275263 (4h50m52.045331504s) [upload=5010]: Got next segment. Inline: true Aug 7 21:17:38.458283519 (4h50m52.04533976s) [upload=5010]: Waiting for error group managing segments... Aug 7 21:17:38.458329695 (4h50m52.045385926s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.458335165 (4h50m52.045391407s) [upload=5010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.458338662 (4h50m52.045394893s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:08.444937554 (4h51m22.031993765s) [upload=5010]: Done uploading segments... Aug 7 21:18:08.444950428 (4h51m22.032006639s) [upload=5010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:50.205377240 (4h52m3.792433451s) [upload=5028]: Starting upload Aug 7 21:18:50.205432072 (4h52m3.792488293s) [upload=5028]: Done starting upload Aug 7 21:18:50.205472938 (4h52m3.792529149s) [upload=5028]: Uploading segments... Aug 7 21:18:50.205500099 (4h52m3.792556311s) [upload=5028]: Got next segment. Inline: true Aug 7 21:18:50.205503937 (4h52m3.792560147s) [upload=5028]: Waiting for error group managing segments... Aug 7 21:18:50.205531809 (4h52m3.79258802s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.205535356 (4h52m3.792591577s) [upload=5028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:50.205538692 (4h52m3.792594913s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:20.193730995 (4h52m33.780787246s) [upload=5028]: Done uploading segments... Aug 7 21:19:20.193750121 (4h52m33.780806372s) [upload=5028]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:03.998495850 (4h53m17.58555209s) [upload=5048]: Starting upload Aug 7 21:20:03.998559819 (4h53m17.585616059s) [upload=5048]: Done starting upload Aug 7 21:20:03.998621224 (4h53m17.585677465s) [upload=5048]: Uploading segments... Aug 7 21:20:03.998661639 (4h53m17.585717879s) [upload=5048]: Got next segment. Inline: true Aug 7 21:20:03.998667419 (4h53m17.58572365s) [upload=5048]: Waiting for error group managing segments... Aug 7 21:20:03.998710790 (4h53m17.585767032s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:03.998716000 (4h53m17.585772242s) [upload=5048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:03.998719587 (4h53m17.585775818s) [upload=5048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:33.984947930 (4h53m47.572004141s) [upload=5048]: Done uploading segments... Aug 7 21:20:33.985025244 (4h53m47.572081455s) [upload=5048]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.0008516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.000881081s FAIL (2025-08-07 21:20:31 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000897541s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 30.001375574s FAIL (2025-08-07 21:20:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001394399s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.001023189s FAIL (2025-08-07 21:20:30 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2776712s) [upload=4897]: Done uploading segments... Aug 7 21:10:58.775736111 (4h44m12.362792352s) [upload=4897]: Upload finished. err: <nil> Aug 7 21:11:31.613155564 (4h44m45.200211785s) [upload=4908]: Starting upload Aug 7 21:11:31.613232477 (4h44m45.200288698s) [upload=4908]: Done starting upload Aug 7 21:11:31.613307799 (4h44m45.20036401s) [upload=4908]: Uploading segments... Aug 7 21:11:31.613347051 (4h44m45.200403272s) [upload=4908]: Got next segment. Inline: true Aug 7 21:11:31.613352932 (4h44m45.200409153s) [upload=4908]: Waiting for error group managing segments... Aug 7 21:11:31.613404179 (4h44m45.2004604s) [upload=4908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.613410591 (4h44m45.200466842s) [upload=4908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.613414508 (4h44m45.200470729s) [upload=4908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.682826735 (4h44m45.269882977s) [upload=4908]: Done uploading segments... Aug 7 21:11:31.682853265 (4h44m45.269909476s) [upload=4908]: Upload finished. err: <nil> Aug 7 21:11:51.754258397 (4h45m5.341314618s) [upload=4915]: Starting upload Aug 7 21:11:51.754330331 (4h45m5.341386542s) [upload=4915]: Done starting upload Aug 7 21:11:51.754383972 (4h45m5.341440193s) [upload=4915]: Uploading segments... Aug 7 21:11:51.754417083 (4h45m5.341473294s) [upload=4915]: Got next segment. Inline: true Aug 7 21:11:51.754420630 (4h45m5.34147684s) [upload=4915]: Waiting for error group managing segments... Aug 7 21:11:51.754453131 (4h45m5.341509352s) [upload=4915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:51.754457108 (4h45m5.341513329s) [upload=4915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:51.754460725 (4h45m5.341516936s) [upload=4915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:51.830658438 (4h45m5.417714649s) [upload=4915]: Done uploading segments... Aug 7 21:11:51.830673094 (4h45m5.417729315s) [upload=4915]: Upload finished. err: <nil> Aug 7 21:12:54.778494646 (4h46m8.365550856s) [upload=4932]: Starting upload Aug 7 21:12:54.778560719 (4h46m8.36561693s) [upload=4932]: Done starting upload Aug 7 21:12:54.778608638 (4h46m8.365664859s) [upload=4932]: Uploading segments... Aug 7 21:12:54.778637322 (4h46m8.365693543s) [upload=4932]: Got next segment. Inline: true Aug 7 21:12:54.778640999 (4h46m8.36569722s) [upload=4932]: Waiting for error group managing segments... Aug 7 21:12:54.778677527 (4h46m8.365733739s) [upload=4932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:54.778681655 (4h46m8.365737876s) [upload=4932]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:54.778684771 (4h46m8.365740983s) [upload=4932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.868239378 (4h46m8.455295599s) [upload=4932]: Done uploading segments... Aug 7 21:12:54.868253865 (4h46m8.455310086s) [upload=4932]: Upload finished. err: <nil> Aug 7 21:13:23.238906839 (4h46m36.82596306s) [upload=4942]: Starting upload Aug 7 21:13:23.238959356 (4h46m36.826015567s) [upload=4942]: Done starting upload Aug 7 21:13:23.239011013 (4h46m36.826067234s) [upload=4942]: Uploading segments... Aug 7 21:13:23.239039727 (4h46m36.826095937s) [upload=4942]: Got next segment. Inline: true Aug 7 21:13:23.239043483 (4h46m36.826099695s) [upload=4942]: Waiting for error group managing segments... Aug 7 21:13:23.239098526 (4h46m36.826154737s) [upload=4942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:23.239104707 (4h46m36.826160919s) [upload=4942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:23.239108474 (4h46m36.826164696s) [upload=4942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:23.309357825 (4h46m36.896414127s) [upload=4942]: Done uploading segments... Aug 7 21:13:23.309383974 (4h46m36.896440216s) [upload=4942]: Upload finished. err: <nil> Aug 7 21:14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:30.042411605 (4h52m43.629467826s) [upload=5040]: Starting upload Aug 7 21:19:30.042491373 (4h52m43.629547594s) [upload=5040]: Done starting upload Aug 7 21:19:30.042544984 (4h52m43.629601195s) [upload=5040]: Uploading segments... Aug 7 21:19:30.042585870 (4h52m43.629642091s) [upload=5040]: Got next segment. Inline: true Aug 7 21:19:30.042590809 (4h52m43.62964703s) [upload=5040]: Waiting for error group managing segments... Aug 7 21:19:30.042637226 (4h52m43.629693436s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:30.042642325 (4h52m43.629698546s) [upload=5040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:30.042647144 (4h52m43.629703365s) [upload=5040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:30.020861737 (4h53m43.607917958s) [upload=5040]: Done uploading segments... Aug 7 21:20:30.020878959 (4h53m43.60793518s) [upload=5040]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001036865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.001048481s FAIL (2025-08-07 21:20:24 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== DJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.288604064 (4h46m37.875660275s) [noise=true , address=104.36.82.23:4457 , piece_id=3LDUSHS23Z6ZBTDHVGVNUZRFA3H2XDC6B243ISKKCYJJG3PCRUBA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.289097404 (4h46m37.876153645s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.303818073 (4h46m37.890874284s) [noise=true , address=104.36.82.20:4444 , piece_id=N3Z2NAQO5LHBPSABVMX4UBAAGUDDGDZCYGWX3NXWP6LUL5VJ74BA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.306609824 (4h46m37.893666035s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.307083188 (4h46m37.894139409s) [noise=true , address=104.36.82.46:4511 , piece_id=XTWLBKT2PX36GPDNTPPQ33HLJE2QMLNTVUMXVDEWJJO6IGC4HCXA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.333002740 (4h46m37.92005898s) [noise=true , address=104.36.82.11:4130 , piece_id=434EKDAN537AQFR342ZKLLENUIZAEK4PL2UTXDRJU7NJJ5HYHLUQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.335712287 (4h46m37.922768508s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.364828819 (4h46m37.95188508s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.415758456 (4h46m38.002814697s) [noise=true , address=104.36.82.37:4680 , piece_id=3LPKEUJ5HCZL4XY7YQOI2CROEHYNBFSZV6EUFDU2WV363FWQ644Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.427338251 (4h46m38.014394462s) [noise=true , address=104.36.82.43:4849 , piece_id=ZVVOBDRSIRLKPAOF23RMYRBXRNCNMQF3TJXYAW77ABOTGYUMXGHQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.490461549 (4h46m38.07751779s) [noise=true , address=79.127.201.215:4354 , piece_id=4VVIMMVTX6SCVPFAODIC7JS5PVULFS2SCXMKFJKA72NXKEK62K3Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.581523329 (4h46m38.16857954s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.581552854 (4h46m38.168609065s) [seg_pos={0 0}, upload=4936]: Segment reached optimal threshold of 14 pieces. Aug 7 21:13:24.581774899 (4h46m38.16883112s) [noise=true , address=79.127.205.233:4643 , piece_id=IRDF6ZKJUCKCPSKAOCHW77H7O5ORLI43QULF2KWJ4O6SJTP373BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cVbdibJiX3T4fEUjeVBNNTTNaMb7m3QYDmxh17Dic5wVUQ9fx): protocol: expected piece hash; context canceled; EOF Aug 7 21:13:24.581831154 (4h46m38.168887375s) [noise=true , address=104.36.82.62:4427 , piece_id=QMUVKOOD3THFKQAKBI3RQWH7JOP34MNNDOMKUXXBF4ZRVSYUJHJQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bx9fnLeXTPskzQB1Ad45qnq4CRivoGy7KXALY35dzVGF6S5Ep): protocol: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection Aug 7 21:13:24.581866029 (4h46m38.168922239s) [noise=true , address=64.49.13.154:4933 , piece_id=HGHDC3TAI5HTPACRHKMJMLCMMF2AZOMT4DO5LIQEIECYYKCHMIRA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n5Mn8FNLXxFWtbbaMv8VuyxuXe6PiiMVtLETzm5aKfxhEykt3): protocol: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection Aug 7 21:13:24.581904450 (4h46m38.168960661s) [noise=true , address=64.49.13.27:4963 , piece_id=MN6MBMOOPEIPZRW5R4NQM6AEE3AHYX33MIXOSIIZEGHDPJEKAOPA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq): protocol: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection Aug 7 21:13:24.581925911 (4h46m38.168982132s) [seg_pos={0 0}, upload=4936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:13:24.582013845 (4h46m38.169070066s) [upload=4936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:24.582018403 (4h46m38.169074624s) [upload=4936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:00.743765287 (4h48m14.330821508s) [upload=4936]: Done uploading segments... Aug 7 21:15:00.743795124 (4h48m14.330851345s) [upload=4936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:00.812344761 (4h48m14.399400973s) [upload=4967]: Starting upload Aug 7 21:15:00.812384405 (4h48m14.399440606s) [upload=4967]: Done starting upload Aug 7 21:15:00.812509518 (4h48m14.399565729s) [upload=4967]: Uploading segments... Aug 7 21:15:00.812592684 (4h48m14.399648895s) [upload=4967]: Got next segment. Inline: false Aug 7 21:15:00.812634802 (4h48m14.399691023s) [upload=4967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:00.812639241 (4h48m14.399695462s) [upload=4967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:00.800009657 (4h49m14.387065878s) [upload=4967]: Next segment err: context deadline exceeded Aug 7 21:16:00.800074909 (4h49m14.38713113s) [upload=4967]: Done uploading segments... Aug 7 21:16:00.800088334 (4h49m14.387144545s) [upload=4967]: Upload finished. err: context deadline exceeded Aug 7 21:16:09.662919397 (4h49m23.249975618s) [upload=4987]: Starting upload Aug 7 21:16:09.663003895 (4h49m23.250060116s) [upload=4987]: Done starting upload Aug 7 21:16:09.663197185 (4h49m23.250253426s) [upload=4987]: Uploading segments... Aug 7 21:16:09.663304947 (4h49m23.250361248s) [upload=4987]: Got next segment. Inline: false Aug 7 21:16:09.663375208 (4h49m23.250431459s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:09.663382832 (4h49m23.250439074s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:09.645661646 (4h50m23.232717887s) [upload=4987]: Next segment err: context deadline exceeded Aug 7 21:17:09.645901063 (4h50m23.232957304s) [upload=4987]: Done uploading segments... Aug 7 21:17:09.645976574 (4h50m23.233032815s) [upload=4987]: Upload finished. err: context deadline exceeded Aug 7 21:17:10.545365898 (4h50m24.132422108s) [upload=5001]: Starting upload Aug 7 21:17:10.545420008 (4h50m24.132476229s) [upload=5001]: Done starting upload Aug 7 21:17:10.545476424 (4h50m24.132532645s) [upload=5001]: Uploading segments... Aug 7 21:17:10.545636733 (4h50m24.132692944s) [upload=5001]: Got next segment. Inline: false Aug 7 21:17:10.545661599 (4h50m24.13271782s) [upload=5001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:10.545665476 (4h50m24.132721688s) [upload=5001]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:10.530620759 (4h51m24.11767701s) [upload=5001]: Next segment err: context deadline exceeded Aug 7 21:18:10.530761893 (4h51m24.117818133s) [upload=5001]: Done uploading segments... Aug 7 21:18:10.530782391 (4h51m24.117838632s) [upload=5001]: Upload finished. err: context deadline exceeded Aug 7 21:18:10.615523121 (4h51m24.202579362s) [upload=5020]: Starting upload Aug 7 21:18:10.615603712 (4h51m24.202659963s) [upload=5020]: Done starting upload Aug 7 21:18:10.615743673 (4h51m24.202799914s) [upload=5020]: Uploading segments... Aug 7 21:18:10.616023836 (4h51m24.203080087s) [upload=5020]: Got next segment. Inline: false Aug 7 21:18:10.616082886 (4h51m24.203139138s) [upload=5020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:10.616091172 (4h51m24.203147413s) [upload=5020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:10.603574083 (4h52m24.190630324s) [upload=5020]: Next segment err: context deadline exceeded Aug 7 21:19:10.603816245 (4h52m24.190872486s) [upload=5020]: Done uploading segments... Aug 7 21:19:10.603869965 (4h52m24.190926207s) [upload=5020]: Upload finished. err: context deadline exceeded Aug 7 21:19:24.417770609 (4h52m38.004826851s) [upload=5037]: Starting upload Aug 7 21:19:24.417854846 (4h52m38.004911088s) [upload=5037]: Done starting upload Aug 7 21:19:24.417989017 (4h52m38.005045258s) [upload=5037]: Uploading segments... Aug 7 21:19:24.418253952 (4h52m38.005310193s) [upload=5037]: Got next segment. Inline: false Aug 7 21:19:24.418302842 (4h52m38.005359084s) [upload=5037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:24.418308183 (4h52m38.005364424s) [upload=5037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:24.399165083 (4h53m37.986221304s) [upload=5037]: Next segment err: context deadline exceeded Aug 7 21:20:24.399217411 (4h53m37.986273622s) [upload=5037]: Done uploading segments... Aug 7 21:20:24.399231628 (4h53m37.986287849s) [upload=5037]: Upload finished. err: context deadline exceeded Duration: 1m0.001067757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 11 out of 60 is failed (threshold=3) [...............................................XX..XXXXXXXXX] duration: 30.000789441s FAIL (2025-08-07 21:20:08 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000812834s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 39.725283ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:20:04 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 39.745531ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 30.00103077s FAIL (2025-08-07 21:20:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001049625s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XX.XXXXX] duration: 1m0.000747616s FAIL (2025-08-07 21:19:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000758566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.262049311s FAIL (2025-08-07 21:19:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 1.262068497s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.000570725s FAIL (2025-08-07 21:19:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000589399s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 26.229712ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:19:37 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 26.252615ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [................................................XX..XXXXXXXX] duration: 30.000918728s FAIL (2025-08-07 21:19:33 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000937432s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 30.000510055s FAIL (2025-08-07 21:19:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000524151s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.001346544s FAIL (2025-08-07 21:19:29 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:10:26.611321150 (4h43m40.198377371s) [upload=4887]: Done uploading segments... Aug 7 21:10:26.611339073 (4h43m40.198395294s) [upload=4887]: Upload finished. err: <nil> Aug 7 21:10:58.699165641 (4h44m12.286221892s) [upload=4897]: Starting upload Aug 7 21:10:58.699245781 (4h44m12.286302022s) [upload=4897]: Done starting upload Aug 7 21:10:58.699314669 (4h44m12.286370911s) [upload=4897]: Uploading segments... Aug 7 21:10:58.699373410 (4h44m12.286429651s) [upload=4897]: Got next segment. Inline: true Aug 7 21:10:58.699380724 (4h44m12.286436964s) [upload=4897]: Waiting for error group managing segments... Aug 7 21:10:58.699426338 (4h44m12.286482579s) [upload=4897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:58.699432380 (4h44m12.28648862s) [upload=4897]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:58.699436066 (4h44m12.286492317s) [upload=4897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:58.775720472 (4h44m12.362776712s) [upload=4897]: Done uploading segments... Aug 7 21:10:58.775736111 (4h44m12.362792352s) [upload=4897]: Upload finished. err: <nil> Aug 7 21:11:31.613155564 (4h44m45.200211785s) [upload=4908]: Starting upload Aug 7 21:11:31.613232477 (4h44m45.200288698s) [upload=4908]: Done starting upload Aug 7 21:11:31.613307799 (4h44m45.20036401s) [upload=4908]: Uploading segments... Aug 7 21:11:31.613347051 (4h44m45.200403272s) [upload=4908]: Got next segment. Inline: true Aug 7 21:11:31.613352932 (4h44m45.200409153s) [upload=4908]: Waiting for error group managing segments... Aug 7 21:11:31.613404179 (4h44m45.2004604s) [upload=4908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.613410591 (4h44m45.200466842s) [upload=4908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.613414508 (4h44m45.200470729s) [upload=4908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.682826735 (4h44m45.269882977s) [upload=4908]: Done uploading segments... Aug 7 21:11:31.682853265 (4h44m45.269909476s) [upload=4908]: Upload finished. err: <nil> Aug 7 21:11:51.754258397 (4h45m5.341314618s) [upload=4915]: Starting upload Aug 7 21:11:51.754330331 (4h45m5.341386542s) [upload=4915]: Done starting upload Aug 7 21:11:51.754383972 (4h45m5.341440193s) [upload=4915]: Uploading segments... Aug 7 21:11:51.754417083 (4h45m5.341473294s) [upload=4915]: Got next segment. Inline: true Aug 7 21:11:51.754420630 (4h45m5.34147684s) [upload=4915]: Waiting for error group managing segments... Aug 7 21:11:51.754453131 (4h45m5.341509352s) [upload=4915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:51.754457108 (4h45m5.341513329s) [upload=4915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:51.754460725 (4h45m5.341516936s) [upload=4915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:51.830658438 (4h45m5.417714649s) [upload=4915]: Done uploading segments... Aug 7 21:11:51.830673094 (4h45m5.417729315s) [upload=4915]: Upload finished. err: <nil> Aug 7 21:12:54.778494646 (4h46m8.365550856s) [upload=4932]: Starting upload Aug 7 21:12:54.778560719 (4h46m8.36561693s) [upload=4932]: Done starting upload Aug 7 21:12:54.778608638 (4h46m8.365664859s) [upload=4932]: Uploading segments... Aug 7 21:12:54.778637322 (4h46m8.365693543s) [upload=4932]: Got next segment. Inline: true Aug 7 21:12:54.778640999 (4h46m8.36569722s) [upload=4932]: Waiting for error group managing segments... Aug 7 21:12:54.778677527 (4h46m8.365733739s) [upload=4932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:54.778681655 (4h46m8.365737876s) [upload=4932]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:54.778684771 (4h46m8.365740983s) [upload=4932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.868239378 (4h46m8.455295599s) [upload=4932]: Done uploading segments... Aug 7 21:12:54.868253865 (4h46m8.455310086s) [upload=4932]: Upload finished. err: <nil> Aug 7 21:13:23.238906839 (4h46m36.82596306s) [upload=4942]: Starting upload Aug 7 21:13:23.238959356 (4h46m36.826015567s) [upload=4942]: Done starting upload Aug 7 21:13:23.239011013 (4h46m36.826067234s) [upload=4942]: Uploading segments... Aug 7 21:13:23.239039727 (4h46m36.826095937s) [upload=4942]: Got next segment. Inline: true Aug 7 21:13:23.239043483 (4h46m36.826099695s) [upload=4942]: Waiting for error group managing segments... Aug 7 21:13:23.239098526 (4h46m36.826154737s) [upload=4942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:23.239104707 (4h46m36.826160919s) [upload=4942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:23.239108474 (4h46m36.826164696s) [upload=4942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:23.309357825 (4h46m36.896414127s) [upload=4942]: Done uploading segments... Aug 7 21:13:23.309383974 (4h46m36.896440216s) [upload=4942]: Upload finished. err: <nil> Aug 7 21:14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:29.988006379 (4h51m43.5750626s) [upload=5023]: Starting upload Aug 7 21:18:29.988053607 (4h51m43.575109818s) [upload=5023]: Done starting upload Aug 7 21:18:29.988109782 (4h51m43.575166003s) [upload=5023]: Uploading segments... Aug 7 21:18:29.988142012 (4h51m43.575198224s) [upload=5023]: Got next segment. Inline: true Aug 7 21:18:29.988147753 (4h51m43.575203973s) [upload=5023]: Waiting for error group managing segments... Aug 7 21:18:29.988197475 (4h51m43.575253687s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:29.988203116 (4h51m43.575259327s) [upload=5023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:29.988206943 (4h51m43.575263164s) [upload=5023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:29.967158594 (4h52m43.554214836s) [upload=5023]: Done uploading segments... Aug 7 21:19:29.967184642 (4h52m43.554240884s) [upload=5023]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.00135987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 30.000428614s FAIL (2025-08-07 21:19:20 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=4903]: Upload finished. err: <nil> Aug 7 21:12:08.543001654 (4h45m22.130057896s) [upload=4919]: Starting upload Aug 7 21:12:08.543123191 (4h45m22.130179432s) [upload=4919]: Done starting upload Aug 7 21:12:08.543216246 (4h45m22.130272487s) [upload=4919]: Uploading segments... Aug 7 21:12:08.543261810 (4h45m22.130318051s) [upload=4919]: Got next segment. Inline: true Aug 7 21:12:08.543267651 (4h45m22.130323892s) [upload=4919]: Waiting for error group managing segments... Aug 7 21:12:08.543313246 (4h45m22.130369488s) [upload=4919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:08.543319278 (4h45m22.130375519s) [upload=4919]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:08.543323075 (4h45m22.130379316s) [upload=4919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:08.630954371 (4h45m22.218010612s) [upload=4919]: Done uploading segments... Aug 7 21:12:08.630979197 (4h45m22.218035449s) [upload=4919]: Upload finished. err: <nil> Aug 7 21:12:30.034507380 (4h45m43.62156362s) [upload=4926]: Starting upload Aug 7 21:12:30.034584302 (4h45m43.621640544s) [upload=4926]: Done starting upload Aug 7 21:12:30.034643523 (4h45m43.621699754s) [upload=4926]: Uploading segments... Aug 7 21:12:30.034681965 (4h45m43.621738206s) [upload=4926]: Got next segment. Inline: true Aug 7 21:12:30.034687645 (4h45m43.621743886s) [upload=4926]: Waiting for error group managing segments... Aug 7 21:12:30.034730766 (4h45m43.621787007s) [upload=4926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:30.034735976 (4h45m43.621792207s) [upload=4926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:30.034739642 (4h45m43.621795883s) [upload=4926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:00.022476261 (4h46m13.609532503s) [upload=4926]: Done uploading segments... Aug 7 21:13:00.022497211 (4h46m13.609553452s) [upload=4926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:15.702096720 (4h46m29.289152941s) [upload=4939]: Starting upload Aug 7 21:13:15.702154518 (4h46m29.289210739s) [upload=4939]: Done starting upload Aug 7 21:13:15.702196716 (4h46m29.289252928s) [upload=4939]: Uploading segments... Aug 7 21:13:15.702224368 (4h46m29.289280589s) [upload=4939]: Got next segment. Inline: true Aug 7 21:13:15.702228446 (4h46m29.289284657s) [upload=4939]: Waiting for error group managing segments... Aug 7 21:13:15.702261458 (4h46m29.289317669s) [upload=4939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:15.702265306 (4h46m29.289321527s) [upload=4939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:15.702267880 (4h46m29.289324101s) [upload=4939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.689162551 (4h46m59.276218792s) [upload=4939]: Done uploading segments... Aug 7 21:13:45.689208046 (4h46m59.276264277s) [upload=4939]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:13.522007492 (4h47m27.109063713s) [upload=4955]: Starting upload Aug 7 21:14:13.522102209 (4h47m27.10915843s) [upload=4955]: Done starting upload Aug 7 21:14:13.522154938 (4h47m27.109211179s) [upload=4955]: Uploading segments... Aug 7 21:14:13.522235488 (4h47m27.109291729s) [upload=4955]: Got next segment. Inline: true Aug 7 21:14:13.522242682 (4h47m27.109298922s) [upload=4955]: Waiting for error group managing segments... Aug 7 21:14:13.522293467 (4h47m27.109349708s) [upload=4955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:13.522299518 (4h47m27.109355759s) [upload=4955]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:13.522303275 (4h47m27.109359516s) [upload=4955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:43.507883903 (4h47m57.094940144s) [upload=4955]: Done uploading segments... Aug 7 21:14:43.507905914 (4h47m57.094962156s) [upload=4955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:54.825501522 (4h48m8.412557734s) [upload=4964]: Starting upload Aug 7 21:14:54.825580730 (4h48m8.412636951s) [upload=4964]: Done starting upload Aug 7 21:14:54.825629411 (4h48m8.412685632s) [upload=4964]: Uploading segments... Aug 7 21:14:54.825660400 (4h48m8.41271661s) [upload=4964]: Got next segment. Inline: true Aug 7 21:14:54.825664547 (4h48m8.412720758s) [upload=4964]: Waiting for error group managing segments... Aug 7 21:14:54.825705873 (4h48m8.412762115s) [upload=4964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:54.825711855 (4h48m8.412768076s) [upload=4964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:54.825715662 (4h48m8.412771883s) [upload=4964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:24.812961328 (4h48m38.400017569s) [upload=4964]: Done uploading segments... Aug 7 21:15:24.812988770 (4h48m38.40004501s) [upload=4964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:40.993035983 (4h48m54.580092194s) [upload=4978]: Starting upload Aug 7 21:15:40.993105403 (4h48m54.580161614s) [upload=4978]: Done starting upload Aug 7 21:15:40.993154163 (4h48m54.580210375s) [upload=4978]: Uploading segments... Aug 7 21:15:40.993189519 (4h48m54.580245741s) [upload=4978]: Got next segment. Inline: true Aug 7 21:15:40.993194319 (4h48m54.58025054s) [upload=4978]: Waiting for error group managing segments... Aug 7 21:15:40.993226930 (4h48m54.580283151s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:40.993231167 (4h48m54.580287379s) [upload=4978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:40.993234264 (4h48m54.580290474s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:10.982081529 (4h49m24.569137771s) [upload=4978]: Done uploading segments... Aug 7 21:16:10.982104473 (4h49m24.569160713s) [upload=4978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:20.227399180 (4h49m33.814455401s) [upload=4990]: Starting upload Aug 7 21:16:20.227450334 (4h49m33.814506545s) [upload=4990]: Done starting upload Aug 7 21:16:20.227491060 (4h49m33.814547272s) [upload=4990]: Uploading segments... Aug 7 21:16:20.227516609 (4h49m33.81457283s) [upload=4990]: Got next segment. Inline: true Aug 7 21:16:20.227520536 (4h49m33.814576757s) [upload=4990]: Waiting for error group managing segments... Aug 7 21:16:20.227552426 (4h49m33.814608636s) [upload=4990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:20.227556263 (4h49m33.814612484s) [upload=4990]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:20.227558527 (4h49m33.814614739s) [upload=4990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:50.216081431 (4h50m3.803137652s) [upload=4990]: Done uploading segments... Aug 7 21:16:50.216097492 (4h50m3.803153713s) [upload=4990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:57.322533914 (4h50m10.909590156s) [upload=4995]: Starting upload Aug 7 21:16:57.322609355 (4h50m10.909665607s) [upload=4995]: Done starting upload Aug 7 21:16:57.322672883 (4h50m10.909729125s) [upload=4995]: Uploading segments... Aug 7 21:16:57.322711135 (4h50m10.909767376s) [upload=4995]: Got next segment. Inline: true Aug 7 21:16:57.322716685 (4h50m10.909772927s) [upload=4995]: Waiting for error group managing segments... Aug 7 21:16:57.322761279 (4h50m10.90981752s) [upload=4995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:57.322767100 (4h50m10.909823341s) [upload=4995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:57.322770697 (4h50m10.909826938s) [upload=4995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:27.310589172 (4h50m40.897645383s) [upload=4995]: Done uploading segments... Aug 7 21:17:27.310603289 (4h50m40.897659499s) [upload=4995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:38.458031408 (4h50m52.045087639s) [upload=5010]: Starting upload Aug 7 21:17:38.458135221 (4h50m52.045191452s) [upload=5010]: Done starting upload Aug 7 21:17:38.458199552 (4h50m52.045255784s) [upload=5010]: Uploading segments... Aug 7 21:17:38.458275263 (4h50m52.045331504s) [upload=5010]: Got next segment. Inline: true Aug 7 21:17:38.458283519 (4h50m52.04533976s) [upload=5010]: Waiting for error group managing segments... Aug 7 21:17:38.458329695 (4h50m52.045385926s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.458335165 (4h50m52.045391407s) [upload=5010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.458338662 (4h50m52.045394893s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:08.444937554 (4h51m22.031993765s) [upload=5010]: Done uploading segments... Aug 7 21:18:08.444950428 (4h51m22.032006639s) [upload=5010]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:50.205377240 (4h52m3.792433451s) [upload=5028]: Starting upload Aug 7 21:18:50.205432072 (4h52m3.792488293s) [upload=5028]: Done starting upload Aug 7 21:18:50.205472938 (4h52m3.792529149s) [upload=5028]: Uploading segments... Aug 7 21:18:50.205500099 (4h52m3.792556311s) [upload=5028]: Got next segment. Inline: true Aug 7 21:18:50.205503937 (4h52m3.792560147s) [upload=5028]: Waiting for error group managing segments... Aug 7 21:18:50.205531809 (4h52m3.79258802s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.205535356 (4h52m3.792591577s) [upload=5028]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:50.205538692 (4h52m3.792594913s) [upload=5028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:20.193730995 (4h52m33.780787246s) [upload=5028]: Done uploading segments... Aug 7 21:19:20.193750121 (4h52m33.780806372s) [upload=5028]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000449283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.000739787s FAIL (2025-08-07 21:19:18 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000769422s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.001136648s FAIL (2025-08-07 21:19:10 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ess=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Uploading piece... Aug 7 21:13:24.169710120 (4h46m37.756766341s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Uploading piece... Aug 7 21:13:24.260747004 (4h46m37.847803255s) [upload=4936]: Next returned nil segment Aug 7 21:13:24.260784884 (4h46m37.847841125s) [upload=4936]: Waiting for error group managing segments... Aug 7 21:13:24.282697478 (4h46m37.869753699s) [noise=true , address=104.36.82.36:4652 , piece_id=4S6GW6S5QMKXREUOZ4K7S77A5FIA5OGIS6MIPH3X5WFAIUYUTKBQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.286011234 (4h46m37.873067475s) [noise=true , address=45.91.116.10:4113 , piece_id=NJYFA7PI7F62J2CSEDJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.288604064 (4h46m37.875660275s) [noise=true , address=104.36.82.23:4457 , piece_id=3LDUSHS23Z6ZBTDHVGVNUZRFA3H2XDC6B243ISKKCYJJG3PCRUBA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.289097404 (4h46m37.876153645s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.303818073 (4h46m37.890874284s) [noise=true , address=104.36.82.20:4444 , piece_id=N3Z2NAQO5LHBPSABVMX4UBAAGUDDGDZCYGWX3NXWP6LUL5VJ74BA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.306609824 (4h46m37.893666035s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.307083188 (4h46m37.894139409s) [noise=true , address=104.36.82.46:4511 , piece_id=XTWLBKT2PX36GPDNTPPQ33HLJE2QMLNTVUMXVDEWJJO6IGC4HCXA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.333002740 (4h46m37.92005898s) [noise=true , address=104.36.82.11:4130 , piece_id=434EKDAN537AQFR342ZKLLENUIZAEK4PL2UTXDRJU7NJJ5HYHLUQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.335712287 (4h46m37.922768508s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.364828819 (4h46m37.95188508s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.415758456 (4h46m38.002814697s) [noise=true , address=104.36.82.37:4680 , piece_id=3LPKEUJ5HCZL4XY7YQOI2CROEHYNBFSZV6EUFDU2WV363FWQ644Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.427338251 (4h46m38.014394462s) [noise=true , address=104.36.82.43:4849 , piece_id=ZVVOBDRSIRLKPAOF23RMYRBXRNCNMQF3TJXYAW77ABOTGYUMXGHQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.490461549 (4h46m38.07751779s) [noise=true , address=79.127.201.215:4354 , piece_id=4VVIMMVTX6SCVPFAODIC7JS5PVULFS2SCXMKFJKA72NXKEK62K3Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.581523329 (4h46m38.16857954s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.581552854 (4h46m38.168609065s) [seg_pos={0 0}, upload=4936]: Segment reached optimal threshold of 14 pieces. Aug 7 21:13:24.581774899 (4h46m38.16883112s) [noise=true , address=79.127.205.233:4643 , piece_id=IRDF6ZKJUCKCPSKAOCHW77H7O5ORLI43QULF2KWJ4O6SJTP373BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cVbdibJiX3T4fEUjeVBNNTTNaMb7m3QYDmxh17Dic5wVUQ9fx): protocol: expected piece hash; context canceled; EOF Aug 7 21:13:24.581831154 (4h46m38.168887375s) [noise=true , address=104.36.82.62:4427 , piece_id=QMUVKOOD3THFKQAKBI3RQWH7JOP34MNNDOMKUXXBF4ZRVSYUJHJQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bx9fnLeXTPskzQB1Ad45qnq4CRivoGy7KXALY35dzVGF6S5Ep): protocol: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection Aug 7 21:13:24.581866029 (4h46m38.168922239s) [noise=true , address=64.49.13.154:4933 , piece_id=HGHDC3TAI5HTPACRHKMJMLCMMF2AZOMT4DO5LIQEIECYYKCHMIRA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n5Mn8FNLXxFWtbbaMv8VuyxuXe6PiiMVtLETzm5aKfxhEykt3): protocol: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection Aug 7 21:13:24.581904450 (4h46m38.168960661s) [noise=true , address=64.49.13.27:4963 , piece_id=MN6MBMOOPEIPZRW5R4NQM6AEE3AHYX33MIXOSIIZEGHDPJEKAOPA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq): protocol: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection Aug 7 21:13:24.581925911 (4h46m38.168982132s) [seg_pos={0 0}, upload=4936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:13:24.582013845 (4h46m38.169070066s) [upload=4936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:24.582018403 (4h46m38.169074624s) [upload=4936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:00.743765287 (4h48m14.330821508s) [upload=4936]: Done uploading segments... Aug 7 21:15:00.743795124 (4h48m14.330851345s) [upload=4936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:00.812344761 (4h48m14.399400973s) [upload=4967]: Starting upload Aug 7 21:15:00.812384405 (4h48m14.399440606s) [upload=4967]: Done starting upload Aug 7 21:15:00.812509518 (4h48m14.399565729s) [upload=4967]: Uploading segments... Aug 7 21:15:00.812592684 (4h48m14.399648895s) [upload=4967]: Got next segment. Inline: false Aug 7 21:15:00.812634802 (4h48m14.399691023s) [upload=4967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:00.812639241 (4h48m14.399695462s) [upload=4967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:00.800009657 (4h49m14.387065878s) [upload=4967]: Next segment err: context deadline exceeded Aug 7 21:16:00.800074909 (4h49m14.38713113s) [upload=4967]: Done uploading segments... Aug 7 21:16:00.800088334 (4h49m14.387144545s) [upload=4967]: Upload finished. err: context deadline exceeded Aug 7 21:16:09.662919397 (4h49m23.249975618s) [upload=4987]: Starting upload Aug 7 21:16:09.663003895 (4h49m23.250060116s) [upload=4987]: Done starting upload Aug 7 21:16:09.663197185 (4h49m23.250253426s) [upload=4987]: Uploading segments... Aug 7 21:16:09.663304947 (4h49m23.250361248s) [upload=4987]: Got next segment. Inline: false Aug 7 21:16:09.663375208 (4h49m23.250431459s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:09.663382832 (4h49m23.250439074s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:09.645661646 (4h50m23.232717887s) [upload=4987]: Next segment err: context deadline exceeded Aug 7 21:17:09.645901063 (4h50m23.232957304s) [upload=4987]: Done uploading segments... Aug 7 21:17:09.645976574 (4h50m23.233032815s) [upload=4987]: Upload finished. err: context deadline exceeded Aug 7 21:17:10.545365898 (4h50m24.132422108s) [upload=5001]: Starting upload Aug 7 21:17:10.545420008 (4h50m24.132476229s) [upload=5001]: Done starting upload Aug 7 21:17:10.545476424 (4h50m24.132532645s) [upload=5001]: Uploading segments... Aug 7 21:17:10.545636733 (4h50m24.132692944s) [upload=5001]: Got next segment. Inline: false Aug 7 21:17:10.545661599 (4h50m24.13271782s) [upload=5001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:10.545665476 (4h50m24.132721688s) [upload=5001]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:10.530620759 (4h51m24.11767701s) [upload=5001]: Next segment err: context deadline exceeded Aug 7 21:18:10.530761893 (4h51m24.117818133s) [upload=5001]: Done uploading segments... Aug 7 21:18:10.530782391 (4h51m24.117838632s) [upload=5001]: Upload finished. err: context deadline exceeded Aug 7 21:18:10.615523121 (4h51m24.202579362s) [upload=5020]: Starting upload Aug 7 21:18:10.615603712 (4h51m24.202659963s) [upload=5020]: Done starting upload Aug 7 21:18:10.615743673 (4h51m24.202799914s) [upload=5020]: Uploading segments... Aug 7 21:18:10.616023836 (4h51m24.203080087s) [upload=5020]: Got next segment. Inline: false Aug 7 21:18:10.616082886 (4h51m24.203139138s) [upload=5020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:10.616091172 (4h51m24.203147413s) [upload=5020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:10.603574083 (4h52m24.190630324s) [upload=5020]: Next segment err: context deadline exceeded Aug 7 21:19:10.603816245 (4h52m24.190872486s) [upload=5020]: Done uploading segments... Aug 7 21:19:10.603869965 (4h52m24.190926207s) [upload=5020]: Upload finished. err: context deadline exceeded Duration: 1m0.001153199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.036362s FAIL (2025-08-07 21:19:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 1.036391235s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.000727199s FAIL (2025-08-07 21:18:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000954955s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XX.XXXX] duration: 1m0.00124501s FAIL (2025-08-07 21:18:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001258795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.000913301s FAIL (2025-08-07 21:18:37 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000931004s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.001050479s FAIL (2025-08-07 21:18:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001065267s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.000982407s FAIL (2025-08-07 21:18:29 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== RequestItem_ObjectCommit Aug 7 21:09:18.367751803 (4h42m31.954808044s) [upload=4869]: Done uploading segments... Aug 7 21:09:18.367774827 (4h42m31.954831078s) [upload=4869]: Upload finished. err: <nil> Aug 7 21:10:26.543295840 (4h43m40.130352072s) [upload=4887]: Starting upload Aug 7 21:10:26.543341496 (4h43m40.130397717s) [upload=4887]: Done starting upload Aug 7 21:10:26.543385698 (4h43m40.130441919s) [upload=4887]: Uploading segments... Aug 7 21:10:26.543414672 (4h43m40.130470883s) [upload=4887]: Got next segment. Inline: true Aug 7 21:10:26.543418339 (4h43m40.13047455s) [upload=4887]: Waiting for error group managing segments... Aug 7 21:10:26.543446882 (4h43m40.130503103s) [upload=4887]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:26.543450790 (4h43m40.130507011s) [upload=4887]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:26.543453054 (4h43m40.130509265s) [upload=4887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:26.611321150 (4h43m40.198377371s) [upload=4887]: Done uploading segments... Aug 7 21:10:26.611339073 (4h43m40.198395294s) [upload=4887]: Upload finished. err: <nil> Aug 7 21:10:58.699165641 (4h44m12.286221892s) [upload=4897]: Starting upload Aug 7 21:10:58.699245781 (4h44m12.286302022s) [upload=4897]: Done starting upload Aug 7 21:10:58.699314669 (4h44m12.286370911s) [upload=4897]: Uploading segments... Aug 7 21:10:58.699373410 (4h44m12.286429651s) [upload=4897]: Got next segment. Inline: true Aug 7 21:10:58.699380724 (4h44m12.286436964s) [upload=4897]: Waiting for error group managing segments... Aug 7 21:10:58.699426338 (4h44m12.286482579s) [upload=4897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:10:58.699432380 (4h44m12.28648862s) [upload=4897]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:10:58.699436066 (4h44m12.286492317s) [upload=4897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:10:58.775720472 (4h44m12.362776712s) [upload=4897]: Done uploading segments... Aug 7 21:10:58.775736111 (4h44m12.362792352s) [upload=4897]: Upload finished. err: <nil> Aug 7 21:11:31.613155564 (4h44m45.200211785s) [upload=4908]: Starting upload Aug 7 21:11:31.613232477 (4h44m45.200288698s) [upload=4908]: Done starting upload Aug 7 21:11:31.613307799 (4h44m45.20036401s) [upload=4908]: Uploading segments... Aug 7 21:11:31.613347051 (4h44m45.200403272s) [upload=4908]: Got next segment. Inline: true Aug 7 21:11:31.613352932 (4h44m45.200409153s) [upload=4908]: Waiting for error group managing segments... Aug 7 21:11:31.613404179 (4h44m45.2004604s) [upload=4908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:31.613410591 (4h44m45.200466842s) [upload=4908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:31.613414508 (4h44m45.200470729s) [upload=4908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:31.682826735 (4h44m45.269882977s) [upload=4908]: Done uploading segments... Aug 7 21:11:31.682853265 (4h44m45.269909476s) [upload=4908]: Upload finished. err: <nil> Aug 7 21:11:51.754258397 (4h45m5.341314618s) [upload=4915]: Starting upload Aug 7 21:11:51.754330331 (4h45m5.341386542s) [upload=4915]: Done starting upload Aug 7 21:11:51.754383972 (4h45m5.341440193s) [upload=4915]: Uploading segments... Aug 7 21:11:51.754417083 (4h45m5.341473294s) [upload=4915]: Got next segment. Inline: true Aug 7 21:11:51.754420630 (4h45m5.34147684s) [upload=4915]: Waiting for error group managing segments... Aug 7 21:11:51.754453131 (4h45m5.341509352s) [upload=4915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:51.754457108 (4h45m5.341513329s) [upload=4915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:51.754460725 (4h45m5.341516936s) [upload=4915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:51.830658438 (4h45m5.417714649s) [upload=4915]: Done uploading segments... Aug 7 21:11:51.830673094 (4h45m5.417729315s) [upload=4915]: Upload finished. err: <nil> Aug 7 21:12:54.778494646 (4h46m8.365550856s) [upload=4932]: Starting upload Aug 7 21:12:54.778560719 (4h46m8.36561693s) [upload=4932]: Done starting upload Aug 7 21:12:54.778608638 (4h46m8.365664859s) [upload=4932]: Uploading segments... Aug 7 21:12:54.778637322 (4h46m8.365693543s) [upload=4932]: Got next segment. Inline: true Aug 7 21:12:54.778640999 (4h46m8.36569722s) [upload=4932]: Waiting for error group managing segments... Aug 7 21:12:54.778677527 (4h46m8.365733739s) [upload=4932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:54.778681655 (4h46m8.365737876s) [upload=4932]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:54.778684771 (4h46m8.365740983s) [upload=4932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:54.868239378 (4h46m8.455295599s) [upload=4932]: Done uploading segments... Aug 7 21:12:54.868253865 (4h46m8.455310086s) [upload=4932]: Upload finished. err: <nil> Aug 7 21:13:23.238906839 (4h46m36.82596306s) [upload=4942]: Starting upload Aug 7 21:13:23.238959356 (4h46m36.826015567s) [upload=4942]: Done starting upload Aug 7 21:13:23.239011013 (4h46m36.826067234s) [upload=4942]: Uploading segments... Aug 7 21:13:23.239039727 (4h46m36.826095937s) [upload=4942]: Got next segment. Inline: true Aug 7 21:13:23.239043483 (4h46m36.826099695s) [upload=4942]: Waiting for error group managing segments... Aug 7 21:13:23.239098526 (4h46m36.826154737s) [upload=4942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:23.239104707 (4h46m36.826160919s) [upload=4942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:23.239108474 (4h46m36.826164696s) [upload=4942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:23.309357825 (4h46m36.896414127s) [upload=4942]: Done uploading segments... Aug 7 21:13:23.309383974 (4h46m36.896440216s) [upload=4942]: Upload finished. err: <nil> Aug 7 21:14:18.294119361 (4h47m31.881175572s) [upload=4957]: Starting upload Aug 7 21:14:18.294187468 (4h47m31.881243679s) [upload=4957]: Done starting upload Aug 7 21:14:18.294261516 (4h47m31.881317737s) [upload=4957]: Uploading segments... Aug 7 21:14:18.294305068 (4h47m31.881361289s) [upload=4957]: Got next segment. Inline: true Aug 7 21:14:18.294311409 (4h47m31.881367621s) [upload=4957]: Waiting for error group managing segments... Aug 7 21:14:18.294349491 (4h47m31.881405712s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:18.294354059 (4h47m31.88141028s) [upload=4957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:18.294356643 (4h47m31.881412854s) [upload=4957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:18.280461255 (4h48m31.867517496s) [upload=4957]: Done uploading segments... Aug 7 21:15:18.280484308 (4h48m31.867540559s) [upload=4957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:18.387536247 (4h48m31.974592488s) [upload=4974]: Starting upload Aug 7 21:15:18.387607770 (4h48m31.974664011s) [upload=4974]: Done starting upload Aug 7 21:15:18.387666109 (4h48m31.974722361s) [upload=4974]: Uploading segments... Aug 7 21:15:18.387706996 (4h48m31.974763236s) [upload=4974]: Got next segment. Inline: true Aug 7 21:15:18.387712766 (4h48m31.974769007s) [upload=4974]: Waiting for error group managing segments... Aug 7 21:15:18.387755796 (4h48m31.974812038s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:18.387761227 (4h48m31.974817468s) [upload=4974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:18.387764853 (4h48m31.974821095s) [upload=4974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:18.373970870 (4h49m31.961027121s) [upload=4974]: Done uploading segments... Aug 7 21:16:18.374035110 (4h49m31.961091352s) [upload=4974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:18.454541239 (4h49m32.04159748s) [upload=4989]: Starting upload Aug 7 21:16:18.454621288 (4h49m32.041677529s) [upload=4989]: Done starting upload Aug 7 21:16:18.454699754 (4h49m32.041756005s) [upload=4989]: Uploading segments... Aug 7 21:16:18.454758153 (4h49m32.041814395s) [upload=4989]: Got next segment. Inline: true Aug 7 21:16:18.454765147 (4h49m32.041821398s) [upload=4989]: Waiting for error group managing segments... Aug 7 21:16:18.454813777 (4h49m32.041870018s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:18.454819818 (4h49m32.04187606s) [upload=4989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:18.454823425 (4h49m32.041879666s) [upload=4989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:18.435319201 (4h50m32.022375413s) [upload=4989]: Done uploading segments... Aug 7 21:17:18.435335041 (4h50m32.022391262s) [upload=4989]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:29.913551414 (4h50m43.500607635s) [upload=5005]: Starting upload Aug 7 21:17:29.913606246 (4h50m43.500662467s) [upload=5005]: Done starting upload Aug 7 21:17:29.913653174 (4h50m43.500709395s) [upload=5005]: Uploading segments... Aug 7 21:17:29.913680986 (4h50m43.500737207s) [upload=5005]: Got next segment. Inline: true Aug 7 21:17:29.913684613 (4h50m43.500740835s) [upload=5005]: Waiting for error group managing segments... Aug 7 21:17:29.913715249 (4h50m43.500771471s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.913719057 (4h50m43.500775278s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:29.913721331 (4h50m43.500777542s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:29.900370534 (4h51m43.487426745s) [upload=5005]: Done uploading segments... Aug 7 21:18:29.900390291 (4h51m43.487446542s) [upload=5005]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000999839s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 32.911804ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:18:24 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 32.936261ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.................................................XX..XXXXXXX] duration: 30.000670156s FAIL (2025-08-07 21:18:19 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000686367s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.00105787s FAIL (2025-08-07 21:18:10 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 21:13:24.168436171 (4h46m37.755492392s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Uploading piece... Aug 7 21:13:24.168641956 (4h46m37.755698177s) [noise=true , address=45.91.116.10:4113 , piece_id=NJYFA7PI7F62J2CSEDJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Uploading piece... Aug 7 21:13:24.168845667 (4h46m37.755901877s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Uploading piece... Aug 7 21:13:24.169076598 (4h46m37.756132819s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Uploading piece... Aug 7 21:13:24.169509506 (4h46m37.756565727s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Uploading piece... Aug 7 21:13:24.169710120 (4h46m37.756766341s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Uploading piece... Aug 7 21:13:24.260747004 (4h46m37.847803255s) [upload=4936]: Next returned nil segment Aug 7 21:13:24.260784884 (4h46m37.847841125s) [upload=4936]: Waiting for error group managing segments... Aug 7 21:13:24.282697478 (4h46m37.869753699s) [noise=true , address=104.36.82.36:4652 , piece_id=4S6GW6S5QMKXREUOZ4K7S77A5FIA5OGIS6MIPH3X5WFAIUYUTKBQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.286011234 (4h46m37.873067475s) [noise=true , address=45.91.116.10:4113 , piece_id=NJYFA7PI7F62J2CSEDJUTIWHY2LW3KBSLK76TKOVJNXC3RDV35KA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.288604064 (4h46m37.875660275s) [noise=true , address=104.36.82.23:4457 , piece_id=3LDUSHS23Z6ZBTDHVGVNUZRFA3H2XDC6B243ISKKCYJJG3PCRUBA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.289097404 (4h46m37.876153645s) [noise=true , address=104.36.82.44:4872 , piece_id=LM4MNB6MDHTN5MVJNGJOWX5Z5AKGICGK5JCLHBQBM5IR5PD3P2BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.303818073 (4h46m37.890874284s) [noise=true , address=104.36.82.20:4444 , piece_id=N3Z2NAQO5LHBPSABVMX4UBAAGUDDGDZCYGWX3NXWP6LUL5VJ74BA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.306609824 (4h46m37.893666035s) [noise=true , address=104.36.82.13:4190 , piece_id=PVB3Y7KXU47SOQCZBKW4AUZ56XMLMJQ23WPM73BKHUMZKARYOCCA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.307083188 (4h46m37.894139409s) [noise=true , address=104.36.82.46:4511 , piece_id=XTWLBKT2PX36GPDNTPPQ33HLJE2QMLNTVUMXVDEWJJO6IGC4HCXA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.333002740 (4h46m37.92005898s) [noise=true , address=104.36.82.11:4130 , piece_id=434EKDAN537AQFR342ZKLLENUIZAEK4PL2UTXDRJU7NJJ5HYHLUQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.335712287 (4h46m37.922768508s) [noise=true , address=79.127.220.97:4396 , piece_id=HD3NDYSYACLTEX2OYRUDYNK5J2DREH3DTNSJSDO745Q6NCC7AXGA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.364828819 (4h46m37.95188508s) [noise=true , address=104.36.82.54:4183 , piece_id=PIBOQ56366UKNWMH43GSS6TW66GFPMSJOB3UQ3TKMEYEYPKVVFOQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.415758456 (4h46m38.002814697s) [noise=true , address=104.36.82.37:4680 , piece_id=3LPKEUJ5HCZL4XY7YQOI2CROEHYNBFSZV6EUFDU2WV363FWQ644Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.427338251 (4h46m38.014394462s) [noise=true , address=104.36.82.43:4849 , piece_id=ZVVOBDRSIRLKPAOF23RMYRBXRNCNMQF3TJXYAW77ABOTGYUMXGHQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.490461549 (4h46m38.07751779s) [noise=true , address=79.127.201.215:4354 , piece_id=4VVIMMVTX6SCVPFAODIC7JS5PVULFS2SCXMKFJKA72NXKEK62K3Q, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.581523329 (4h46m38.16857954s) [noise=true , address=104.36.82.52:4108 , piece_id=AHQKVW323AG3TIIDCPX4HYFISVIY4V2476ZQM5GXLGLQD2PJ2KVA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: <nil> Aug 7 21:13:24.581552854 (4h46m38.168609065s) [seg_pos={0 0}, upload=4936]: Segment reached optimal threshold of 14 pieces. Aug 7 21:13:24.581774899 (4h46m38.16883112s) [noise=true , address=79.127.205.233:4643 , piece_id=IRDF6ZKJUCKCPSKAOCHW77H7O5ORLI43QULF2KWJ4O6SJTP373BQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cVbdibJiX3T4fEUjeVBNNTTNaMb7m3QYDmxh17Dic5wVUQ9fx): protocol: expected piece hash; context canceled; EOF Aug 7 21:13:24.581831154 (4h46m38.168887375s) [noise=true , address=104.36.82.62:4427 , piece_id=QMUVKOOD3THFKQAKBI3RQWH7JOP34MNNDOMKUXXBF4ZRVSYUJHJQ, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bx9fnLeXTPskzQB1Ad45qnq4CRivoGy7KXALY35dzVGF6S5Ep): protocol: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41822->104.36.82.62:4427: use of closed network connection Aug 7 21:13:24.581866029 (4h46m38.168922239s) [noise=true , address=64.49.13.154:4933 , piece_id=HGHDC3TAI5HTPACRHKMJMLCMMF2AZOMT4DO5LIQEIECYYKCHMIRA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n5Mn8FNLXxFWtbbaMv8VuyxuXe6PiiMVtLETzm5aKfxhEykt3): protocol: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43274->64.49.13.154:4933: use of closed network connection Aug 7 21:13:24.581904450 (4h46m38.168960661s) [noise=true , address=64.49.13.27:4963 , piece_id=MN6MBMOOPEIPZRW5R4NQM6AEE3AHYX33MIXOSIIZEGHDPJEKAOPA, seg_pos={0 0}, upload=4936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq): protocol: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54020->64.49.13.27:4963: use of closed network connection Aug 7 21:13:24.581925911 (4h46m38.168982132s) [seg_pos={0 0}, upload=4936]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:13:24.582013845 (4h46m38.169070066s) [upload=4936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:24.582018403 (4h46m38.169074624s) [upload=4936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:00.743765287 (4h48m14.330821508s) [upload=4936]: Done uploading segments... Aug 7 21:15:00.743795124 (4h48m14.330851345s) [upload=4936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:00.812344761 (4h48m14.399400973s) [upload=4967]: Starting upload Aug 7 21:15:00.812384405 (4h48m14.399440606s) [upload=4967]: Done starting upload Aug 7 21:15:00.812509518 (4h48m14.399565729s) [upload=4967]: Uploading segments... Aug 7 21:15:00.812592684 (4h48m14.399648895s) [upload=4967]: Got next segment. Inline: false Aug 7 21:15:00.812634802 (4h48m14.399691023s) [upload=4967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:00.812639241 (4h48m14.399695462s) [upload=4967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:00.800009657 (4h49m14.387065878s) [upload=4967]: Next segment err: context deadline exceeded Aug 7 21:16:00.800074909 (4h49m14.38713113s) [upload=4967]: Done uploading segments... Aug 7 21:16:00.800088334 (4h49m14.387144545s) [upload=4967]: Upload finished. err: context deadline exceeded Aug 7 21:16:09.662919397 (4h49m23.249975618s) [upload=4987]: Starting upload Aug 7 21:16:09.663003895 (4h49m23.250060116s) [upload=4987]: Done starting upload Aug 7 21:16:09.663197185 (4h49m23.250253426s) [upload=4987]: Uploading segments... Aug 7 21:16:09.663304947 (4h49m23.250361248s) [upload=4987]: Got next segment. Inline: false Aug 7 21:16:09.663375208 (4h49m23.250431459s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:09.663382832 (4h49m23.250439074s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:09.645661646 (4h50m23.232717887s) [upload=4987]: Next segment err: context deadline exceeded Aug 7 21:17:09.645901063 (4h50m23.232957304s) [upload=4987]: Done uploading segments... Aug 7 21:17:09.645976574 (4h50m23.233032815s) [upload=4987]: Upload finished. err: context deadline exceeded Aug 7 21:17:10.545365898 (4h50m24.132422108s) [upload=5001]: Starting upload Aug 7 21:17:10.545420008 (4h50m24.132476229s) [upload=5001]: Done starting upload Aug 7 21:17:10.545476424 (4h50m24.132532645s) [upload=5001]: Uploading segments... Aug 7 21:17:10.545636733 (4h50m24.132692944s) [upload=5001]: Got next segment. Inline: false Aug 7 21:17:10.545661599 (4h50m24.13271782s) [upload=5001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:10.545665476 (4h50m24.132721688s) [upload=5001]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:10.530620759 (4h51m24.11767701s) [upload=5001]: Next segment err: context deadline exceeded Aug 7 21:18:10.530761893 (4h51m24.117818133s) [upload=5001]: Done uploading segments... Aug 7 21:18:10.530782391 (4h51m24.117838632s) [upload=5001]: Upload finished. err: context deadline exceeded Duration: 1m0.001077075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 30.001058935s FAIL (2025-08-07 21:18:08 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (4h44m2.623148821s) [upload=4895]: Upload finished. err: <nil> Aug 7 21:11:10.299224768 (4h44m23.886281009s) [upload=4903]: Starting upload Aug 7 21:11:10.299304937 (4h44m23.886361178s) [upload=4903]: Done starting upload Aug 7 21:11:10.299368445 (4h44m23.886424686s) [upload=4903]: Uploading segments... Aug 7 21:11:10.299409052 (4h44m23.886465292s) [upload=4903]: Got next segment. Inline: true Aug 7 21:11:10.299414652 (4h44m23.886470893s) [upload=4903]: Waiting for error group managing segments... Aug 7 21:11:10.299460468 (4h44m23.886516709s) [upload=4903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:11:10.299465988 (4h44m23.886522229s) [upload=4903]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:11:10.299469444 (4h44m23.886525685s) [upload=4903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:11:10.444467755 (4h44m24.031523996s) [upload=4903]: Done uploading segments... Aug 7 21:11:10.444485819 (4h44m24.03154206s) [upload=4903]: Upload finished. err: <nil> Aug 7 21:12:08.543001654 (4h45m22.130057896s) [upload=4919]: Starting upload Aug 7 21:12:08.543123191 (4h45m22.130179432s) [upload=4919]: Done starting upload Aug 7 21:12:08.543216246 (4h45m22.130272487s) [upload=4919]: Uploading segments... Aug 7 21:12:08.543261810 (4h45m22.130318051s) [upload=4919]: Got next segment. Inline: true Aug 7 21:12:08.543267651 (4h45m22.130323892s) [upload=4919]: Waiting for error group managing segments... Aug 7 21:12:08.543313246 (4h45m22.130369488s) [upload=4919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:08.543319278 (4h45m22.130375519s) [upload=4919]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:08.543323075 (4h45m22.130379316s) [upload=4919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:12:08.630954371 (4h45m22.218010612s) [upload=4919]: Done uploading segments... Aug 7 21:12:08.630979197 (4h45m22.218035449s) [upload=4919]: Upload finished. err: <nil> Aug 7 21:12:30.034507380 (4h45m43.62156362s) [upload=4926]: Starting upload Aug 7 21:12:30.034584302 (4h45m43.621640544s) [upload=4926]: Done starting upload Aug 7 21:12:30.034643523 (4h45m43.621699754s) [upload=4926]: Uploading segments... Aug 7 21:12:30.034681965 (4h45m43.621738206s) [upload=4926]: Got next segment. Inline: true Aug 7 21:12:30.034687645 (4h45m43.621743886s) [upload=4926]: Waiting for error group managing segments... Aug 7 21:12:30.034730766 (4h45m43.621787007s) [upload=4926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:12:30.034735976 (4h45m43.621792207s) [upload=4926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:30.034739642 (4h45m43.621795883s) [upload=4926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:00.022476261 (4h46m13.609532503s) [upload=4926]: Done uploading segments... Aug 7 21:13:00.022497211 (4h46m13.609553452s) [upload=4926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:13:15.702096720 (4h46m29.289152941s) [upload=4939]: Starting upload Aug 7 21:13:15.702154518 (4h46m29.289210739s) [upload=4939]: Done starting upload Aug 7 21:13:15.702196716 (4h46m29.289252928s) [upload=4939]: Uploading segments... Aug 7 21:13:15.702224368 (4h46m29.289280589s) [upload=4939]: Got next segment. Inline: true Aug 7 21:13:15.702228446 (4h46m29.289284657s) [upload=4939]: Waiting for error group managing segments... Aug 7 21:13:15.702261458 (4h46m29.289317669s) [upload=4939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:13:15.702265306 (4h46m29.289321527s) [upload=4939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:15.702267880 (4h46m29.289324101s) [upload=4939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:45.689162551 (4h46m59.276218792s) [upload=4939]: Done uploading segments... Aug 7 21:13:45.689208046 (4h46m59.276264277s) [upload=4939]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:13.522007492 (4h47m27.109063713s) [upload=4955]: Starting upload Aug 7 21:14:13.522102209 (4h47m27.10915843s) [upload=4955]: Done starting upload Aug 7 21:14:13.522154938 (4h47m27.109211179s) [upload=4955]: Uploading segments... Aug 7 21:14:13.522235488 (4h47m27.109291729s) [upload=4955]: Got next segment. Inline: true Aug 7 21:14:13.522242682 (4h47m27.109298922s) [upload=4955]: Waiting for error group managing segments... Aug 7 21:14:13.522293467 (4h47m27.109349708s) [upload=4955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:13.522299518 (4h47m27.109355759s) [upload=4955]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:13.522303275 (4h47m27.109359516s) [upload=4955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:43.507883903 (4h47m57.094940144s) [upload=4955]: Done uploading segments... Aug 7 21:14:43.507905914 (4h47m57.094962156s) [upload=4955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:14:54.825501522 (4h48m8.412557734s) [upload=4964]: Starting upload Aug 7 21:14:54.825580730 (4h48m8.412636951s) [upload=4964]: Done starting upload Aug 7 21:14:54.825629411 (4h48m8.412685632s) [upload=4964]: Uploading segments... Aug 7 21:14:54.825660400 (4h48m8.41271661s) [upload=4964]: Got next segment. Inline: true Aug 7 21:14:54.825664547 (4h48m8.412720758s) [upload=4964]: Waiting for error group managing segments... Aug 7 21:14:54.825705873 (4h48m8.412762115s) [upload=4964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:54.825711855 (4h48m8.412768076s) [upload=4964]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:54.825715662 (4h48m8.412771883s) [upload=4964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:24.812961328 (4h48m38.400017569s) [upload=4964]: Done uploading segments... Aug 7 21:15:24.812988770 (4h48m38.40004501s) [upload=4964]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:40.993035983 (4h48m54.580092194s) [upload=4978]: Starting upload Aug 7 21:15:40.993105403 (4h48m54.580161614s) [upload=4978]: Done starting upload Aug 7 21:15:40.993154163 (4h48m54.580210375s) [upload=4978]: Uploading segments... Aug 7 21:15:40.993189519 (4h48m54.580245741s) [upload=4978]: Got next segment. Inline: true Aug 7 21:15:40.993194319 (4h48m54.58025054s) [upload=4978]: Waiting for error group managing segments... Aug 7 21:15:40.993226930 (4h48m54.580283151s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:40.993231167 (4h48m54.580287379s) [upload=4978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:40.993234264 (4h48m54.580290474s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:10.982081529 (4h49m24.569137771s) [upload=4978]: Done uploading segments... Aug 7 21:16:10.982104473 (4h49m24.569160713s) [upload=4978]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:20.227399180 (4h49m33.814455401s) [upload=4990]: Starting upload Aug 7 21:16:20.227450334 (4h49m33.814506545s) [upload=4990]: Done starting upload Aug 7 21:16:20.227491060 (4h49m33.814547272s) [upload=4990]: Uploading segments... Aug 7 21:16:20.227516609 (4h49m33.81457283s) [upload=4990]: Got next segment. Inline: true Aug 7 21:16:20.227520536 (4h49m33.814576757s) [upload=4990]: Waiting for error group managing segments... Aug 7 21:16:20.227552426 (4h49m33.814608636s) [upload=4990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:20.227556263 (4h49m33.814612484s) [upload=4990]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:20.227558527 (4h49m33.814614739s) [upload=4990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:50.216081431 (4h50m3.803137652s) [upload=4990]: Done uploading segments... Aug 7 21:16:50.216097492 (4h50m3.803153713s) [upload=4990]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:57.322533914 (4h50m10.909590156s) [upload=4995]: Starting upload Aug 7 21:16:57.322609355 (4h50m10.909665607s) [upload=4995]: Done starting upload Aug 7 21:16:57.322672883 (4h50m10.909729125s) [upload=4995]: Uploading segments... Aug 7 21:16:57.322711135 (4h50m10.909767376s) [upload=4995]: Got next segment. Inline: true Aug 7 21:16:57.322716685 (4h50m10.909772927s) [upload=4995]: Waiting for error group managing segments... Aug 7 21:16:57.322761279 (4h50m10.90981752s) [upload=4995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:57.322767100 (4h50m10.909823341s) [upload=4995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:57.322770697 (4h50m10.909826938s) [upload=4995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:27.310589172 (4h50m40.897645383s) [upload=4995]: Done uploading segments... Aug 7 21:17:27.310603289 (4h50m40.897659499s) [upload=4995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:38.458031408 (4h50m52.045087639s) [upload=5010]: Starting upload Aug 7 21:17:38.458135221 (4h50m52.045191452s) [upload=5010]: Done starting upload Aug 7 21:17:38.458199552 (4h50m52.045255784s) [upload=5010]: Uploading segments... Aug 7 21:17:38.458275263 (4h50m52.045331504s) [upload=5010]: Got next segment. Inline: true Aug 7 21:17:38.458283519 (4h50m52.04533976s) [upload=5010]: Waiting for error group managing segments... Aug 7 21:17:38.458329695 (4h50m52.045385926s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.458335165 (4h50m52.045391407s) [upload=5010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.458338662 (4h50m52.045394893s) [upload=5010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:08.444937554 (4h51m22.031993765s) [upload=5010]: Done uploading segments... Aug 7 21:18:08.444950428 (4h51m22.032006639s) [upload=5010]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00108324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.001150235s FAIL (2025-08-07 21:18:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001165662s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.014645574s FAIL (2025-08-07 21:18:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 1.014678165s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.000770816s FAIL (2025-08-07 21:17:59 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== .82.9:4006 , piece_id=EH5ZUHUO55F7YYVD2TSWXDIUX65GY2VVVMBW5DOFRADL3WO3ZQGA, seg_pos={0 0}, upload=4938]: Uploading piece... Aug 7 21:13:09.841271227 (4h46m23.428327438s) [upload=4938]: Next returned nil segment Aug 7 21:13:09.841281055 (4h46m23.428337266s) [upload=4938]: Waiting for error group managing segments... Aug 7 21:13:09.853631449 (4h46m23.440687659s) [noise=true , address=45.91.116.4:4937 , piece_id=HUS3E4LN6CYOXDY2DIEK4OATMZPZADDVIQO3CM3L5QEMB567P3KA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.854204258 (4h46m23.441260499s) [noise=true , address=45.91.116.12:4225 , piece_id=EPHX3ZYTEBJUTQP56HQSL4GPW5SBMC7KDNSR6BD7CRQTMCDUZ2DQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.854658546 (4h46m23.441714757s) [noise=true , address=45.91.116.14:4191 , piece_id=23HFDLS3OMLHX4EBBB3N6ORXPOJGZ4TSLXD622Z5QW57I7F33YSQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.855360527 (4h46m23.442416778s) [noise=true , address=77.90.31.18:4807 , piece_id=GYJGSHU2PC4DM5VA6SLHGM7S45KOLLBLBTMKD6LT5LMAEMHKGK5Q, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.855562183 (4h46m23.442618394s) [noise=true , address=104.36.82.13:4185 , piece_id=JRWMK5KTTCJIL5JJ4YVJ3GHS6BYCZP3CRWM4AN4ZL2UU4HJADBSQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.856428770 (4h46m23.443485021s) [noise=true , address=77.90.31.19:4841 , piece_id=AKJV4UNYIXKLSI4Y5STPWBXJ3AFZERBSVCIER53F23KL5WABQFTQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.856590482 (4h46m23.443646684s) [noise=true , address=104.36.82.2:4729 , piece_id=KQOVEHNICVG5FWPV2B7CP3U6SMP56T7PRNZOG65QDAX2MP4SEN7Q, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.858568354 (4h46m23.445624565s) [noise=true , address=45.91.116.13:4155 , piece_id=BXSMK5DW67MELF2KN6CIDHJ5HOSF7MZBRVSA7WQ45UV4SLI5XHWQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.858850201 (4h46m23.445906441s) [noise=true , address=77.90.31.16:4733 , piece_id=EMDHMK2PGYUT3SUXOVNEKLRHOH3XY4WNXON4OXTERRMZI43GCROA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.859375651 (4h46m23.446431862s) [noise=true , address=77.90.31.17:4772 , piece_id=QUJ2AZMAXL2FFEWVASVIN734LGSMBFRXHX5KORJLQNGIXNN64ZVA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.859727598 (4h46m23.446783819s) [noise=true , address=104.36.82.13:4187 , piece_id=J3E4KCSQZDYKXQMOTRSUUUIU2SZIRFBM7WD32FCFJL7V7DILAQXA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.862181970 (4h46m23.44923821s) [noise=true , address=104.36.82.12:4151 , piece_id=OIWOXCCCVAW3B5ISSLNWASC6TD2WSWC4J3EJMF554OHVQMCZUIWQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.862286064 (4h46m23.449342285s) [noise=true , address=104.36.82.7:4910 , piece_id=HFOIPKET4SKDQ4LRX54T7VOA7V2Y46MZ6RH5DED63ZXTTDYZTHQA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: <nil> Aug 7 21:13:09.862295481 (4h46m23.449351692s) [seg_pos={0 0}, upload=4938]: Segment reached optimal threshold of 13 pieces. Aug 7 21:13:09.862540559 (4h46m23.44959677s) [noise=true , address=104.36.82.9:4006 , piece_id=EH5ZUHUO55F7YYVD2TSWXDIUX65GY2VVVMBW5DOFRADL3WO3ZQGA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection; write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:35382->104.36.82.9:4006: use of closed network connection Aug 7 21:13:09.862594711 (4h46m23.449650932s) [noise=true , address=104.36.82.5:4839 , piece_id=JFXBI2UEF5IPMF3NFKEX4CBRRNCIBJCSLLSVKCTFRTYRD3U4TLLQ, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection; write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:52042->104.36.82.5:4839: use of closed network connection Aug 7 21:13:09.862660843 (4h46m23.449717085s) [noise=true , address=104.36.82.15:4259 , piece_id=AZDG4LGBEPEVUA4KZH24JWD2BCFEUA2IGBFWC57N4D5WVXBJ27PA, seg_pos={0 0}, upload=4938]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection; write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8524->104.36.82.15:4259: use of closed network connection Aug 7 21:13:09.862727478 (4h46m23.449783718s) [seg_pos={0 0}, upload=4938]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:13:09.862892626 (4h46m23.449948867s) [upload=4938]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:13:09.862899970 (4h46m23.449956211s) [upload=4938]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:13:09.912447959 (4h46m23.499504169s) [upload=4938]: Done uploading segments... Aug 7 21:13:09.912480760 (4h46m23.499536981s) [upload=4938]: Upload finished. err: <nil> Aug 7 21:14:19.537010001 (4h47m33.124066212s) [upload=4958]: Starting upload Aug 7 21:14:19.537096253 (4h47m33.124152504s) [upload=4958]: Done starting upload Aug 7 21:14:19.537202491 (4h47m33.124258732s) [upload=4958]: Uploading segments... Aug 7 21:14:19.537385743 (4h47m33.124441964s) [upload=4958]: Got next segment. Inline: false Aug 7 21:14:19.537463918 (4h47m33.124520139s) [upload=4958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:19.537469729 (4h47m33.12452595s) [upload=4958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:49.522961209 (4h48m3.11001742s) [upload=4958]: Next segment err: context deadline exceeded Aug 7 21:14:49.523117610 (4h48m3.110173821s) [upload=4958]: Done uploading segments... Aug 7 21:14:49.523153057 (4h48m3.110209278s) [upload=4958]: Upload finished. err: context deadline exceeded Aug 7 21:15:15.778236986 (4h48m29.365293237s) [upload=4971]: Starting upload Aug 7 21:15:15.778363202 (4h48m29.365419443s) [upload=4971]: Done starting upload Aug 7 21:15:15.778550181 (4h48m29.365606442s) [upload=4971]: Uploading segments... Aug 7 21:15:15.778800839 (4h48m29.36585709s) [upload=4971]: Got next segment. Inline: false Aug 7 21:15:15.778897088 (4h48m29.36595334s) [upload=4971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:15.778908650 (4h48m29.365964891s) [upload=4971]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:45.754152630 (4h48m59.341208851s) [upload=4971]: Next segment err: context deadline exceeded Aug 7 21:15:45.754250592 (4h48m59.341306803s) [upload=4971]: Done uploading segments... Aug 7 21:15:45.754272974 (4h48m59.341329186s) [upload=4971]: Upload finished. err: context deadline exceeded Aug 7 21:15:54.388794729 (4h49m7.975850951s) [upload=4981]: Starting upload Aug 7 21:15:54.388846236 (4h49m7.975902457s) [upload=4981]: Done starting upload Aug 7 21:15:54.388936303 (4h49m7.975992524s) [upload=4981]: Uploading segments... Aug 7 21:15:54.389106811 (4h49m7.976163032s) [upload=4981]: Got next segment. Inline: false Aug 7 21:15:54.389136207 (4h49m7.976192428s) [upload=4981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:54.389140785 (4h49m7.976197006s) [upload=4981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:24.377635868 (4h49m37.96469211s) [upload=4981]: Next segment err: context deadline exceeded Aug 7 21:16:24.377706209 (4h49m37.96476246s) [upload=4981]: Done uploading segments... Aug 7 21:16:24.377728742 (4h49m37.964784992s) [upload=4981]: Upload finished. err: context deadline exceeded Aug 7 21:16:48.671509340 (4h50m2.258565551s) [upload=4993]: Starting upload Aug 7 21:16:48.671596032 (4h50m2.258652243s) [upload=4993]: Done starting upload Aug 7 21:16:48.671714142 (4h50m2.258770383s) [upload=4993]: Uploading segments... Aug 7 21:16:48.671899698 (4h50m2.25895594s) [upload=4993]: Got next segment. Inline: false Aug 7 21:16:48.671982002 (4h50m2.259038243s) [upload=4993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:48.671989065 (4h50m2.259045306s) [upload=4993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:18.658253100 (4h50m32.245309321s) [upload=4993]: Next segment err: context deadline exceeded Aug 7 21:17:18.658397720 (4h50m32.245453941s) [upload=4993]: Done uploading segments... Aug 7 21:17:18.658431633 (4h50m32.245487854s) [upload=4993]: Upload finished. err: context deadline exceeded Aug 7 21:17:29.086779152 (4h50m42.673835373s) [upload=5004]: Starting upload Aug 7 21:17:29.086833233 (4h50m42.673889444s) [upload=5004]: Done starting upload Aug 7 21:17:29.086916819 (4h50m42.67397304s) [upload=5004]: Uploading segments... Aug 7 21:17:29.087068823 (4h50m42.674125034s) [upload=5004]: Got next segment. Inline: false Aug 7 21:17:29.087119998 (4h50m42.674176219s) [upload=5004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:29.087124536 (4h50m42.674180748s) [upload=5004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:59.072413634 (4h51m12.659469875s) [upload=5004]: Next segment err: context deadline exceeded Aug 7 21:17:59.072514252 (4h51m12.659570503s) [upload=5004]: Done uploading segments... Aug 7 21:17:59.072534319 (4h51m12.659590571s) [upload=5004]: Upload finished. err: context deadline exceeded Duration: 30.00078927s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.001216165s FAIL (2025-08-07 21:17:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001227518s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.001247535s FAIL (2025-08-07 21:17:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001266729s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XX.XXX] duration: 1m0.000493193s FAIL (2025-08-07 21:17:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000506988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..................................................XX..XXXXXX] duration: 30.00052132s FAIL (2025-08-07 21:17:49 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000545607s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 563.883791ms FAIL (2025-08-07 21:17:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.601346920 (4h35m2.188403141s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602014457 (4h35m2.189070678s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602175055 (4h35m2.189231277s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602655432 (4h35m2.189711644s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.602849785 (4h35m2.189906006s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603565100 (4h35m2.190621321s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603577724 (4h35m2.190633945s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603630644 (4h35m2.190686855s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.603928529 (4h35m2.19098475s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604153980 (4h35m2.191210201s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604387185 (4h35m2.191443406s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604744643 (4h35m2.191800864s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.604931691 (4h35m2.191987903s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.605523377 (4h35m2.192579598s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.609320384 (4h35m2.196376595s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Uploading piece... Aug 7 21:01:48.674305087 (4h35m2.261361299s) [upload=4741]: Next returned nil segment Aug 7 21:01:48.674322240 (4h35m2.261378451s) [upload=4741]: Waiting for error group managing segments... Aug 7 21:01:48.698882585 (4h35m2.285938796s) [noise=true , address=45.91.116.13:4184 , piece_id=CYVUHYB7MB3C5L3IA2OSA5X5LMAU4Z5I3PBKBQRTZJGY5RPEUTVA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.702260020 (4h35m2.289316221s) [noise=true , address=104.36.82.61:4375 , piece_id=EVGZEVJFKBR26BCJZVVRYQPROICZPMZUQRG6TXGKQ5MIOS2U33KQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.709310800 (4h35m2.296367012s) [noise=true , address=104.36.82.15:4273 , piece_id=E5KMED3OODLLOMH772ZLY3KPV2QKBJMEMZUKTXWZBDJVFB5PQQXQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.710480384 (4h35m2.297536605s) [noise=true , address=104.36.82.35:4582 , piece_id=7XPBU22P4VL3FOX22UCTPDSZZWR6Z2QYPEDQ5645KZPZHARKPG6Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.726552304 (4h35m2.313608525s) [noise=true , address=104.36.82.38:4711 , piece_id=EK4ORPEZSKF2LBMW5XK7FTXV5LILM4LWKN5IJDFIOXFQZ46RRJ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.735555450 (4h35m2.322611661s) [noise=true , address=104.36.82.44:4873 , piece_id=X2IB7FNHLTJ6DFRCOS4MXGOSO3NBIGPA4ZV6GNHUWPV5AAINRZ4A, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.748292316 (4h35m2.335348527s) [noise=true , address=104.36.82.60:4360 , piece_id=PBAR27QDYDYE43UYKS4EQKKVF6QZZ42UMWFBBKF7YI5HWI34GFHA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.749773620 (4h35m2.336829832s) [noise=true , address=104.36.82.13:4207 , piece_id=PTURKUKGP32DRRYU2FJCDGTYNQUPPC3HURKZX4HX52R4RKKIRD4Q, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750299954 (4h35m2.337356165s) [noise=true , address=104.36.82.23:4447 , piece_id=6ASIGUT3Z4QFCR43JOL3XLVQ5ACHTCXN7CTPOFHMLQXZ52H2A6MA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.750910052 (4h35m2.337966303s) [noise=true , address=45.91.116.7:4032 , piece_id=PYPRNETMVWPIODOI5TYMML5ULQ52ABM3UYCJJYBP7JJOUW6PNTRQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.768808221 (4h35m2.355864443s) [noise=true , address=104.36.82.4:4825 , piece_id=H7IXZDO3RMPEYP5X632YGC4IFAG6GRDAKGFXK2ULBOTKX5EPFSLA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.810369602 (4h35m2.397425823s) [noise=true , address=104.36.82.39:4753 , piece_id=2QSICT65DH3ZRGVW3JHBOPAIP737YVXFTLP6SMY2LDVQ727OQPUQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:48.840971094 (4h35m2.428027305s) [noise=true , address=104.36.82.43:4856 , piece_id=KMCGUX5ERXTCLTDIVLQ4RO7VJN2ME2PWLRNNRKUEJ2SPCD5MEGWQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007362780 (4h35m2.594418991s) [noise=true , address=104.36.82.12:4155 , piece_id=KU7AIMH6HWK2J3UUS5G2GUSHU6REMKAWSLMKZSLMK4Z4NQOBVOKQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: <nil> Aug 7 21:01:49.007381163 (4h35m2.594437374s) [seg_pos={0 0}, upload=4741]: Segment reached optimal threshold of 14 pieces. Aug 7 21:01:49.007613637 (4h35m2.594669848s) [noise=true , address=104.36.82.47:4525 , piece_id=HASUEPAY5OWMAHFKWGXYPWYWYRIR2LXSCRKYGGNFDE5KJJ5X4LFQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX): protocol: expected piece hash; context canceled; EOF Aug 7 21:01:49.007678349 (4h35m2.59473456s) [noise=true , address=193.149.164.143:4696 , piece_id=3OOJ6XGMU6Y3MTPFIFSNZYNS7SMXCTPPGVHOU7HRX5FXWL4I4TCQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56072->193.149.164.143:4696: use of closed network connection Aug 7 21:01:49.007766854 (4h35m2.594823065s) [noise=true , address=104.36.82.58:4271 , piece_id=HC4JBQXYRY2NE3M7JCA2WYZWVMWQSMYFBHGIYRNKVD4GNAIWVKZA, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: read tcp 5.161.89.252:21134->104.36.82.58:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:01:49.007891316 (4h35m2.594947527s) [noise=true , address=104.36.82.59:4331 , piece_id=OIXPWBMN5TUCVGF4J636JFP6DIUWDRGL7E2SM44PU3FQB44U32CQ, seg_pos={0 0}, upload=4741]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61270->104.36.82.59:4331: use of closed network connection Aug 7 21:01:49.007944575 (4h35m2.595000796s) [seg_pos={0 0}, upload=4741]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:01:49.008086300 (4h35m2.595142521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:01:49.008091300 (4h35m2.595147521s) [upload=4741]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:01:49.089771998 (4h35m2.676828219s) [upload=4741]: Done uploading segments... Aug 7 21:01:49.089786797 (4h35m2.676843028s) [upload=4741]: Upload finished. err: <nil> Aug 7 21:02:04.233419851 (4h35m17.820476092s) [upload=4749]: Starting upload Aug 7 21:02:04.233540527 (4h35m17.820596767s) [upload=4749]: Done starting upload Aug 7 21:02:04.233782818 (4h35m17.820839039s) [upload=4749]: Uploading segments... Aug 7 21:02:04.233984515 (4h35m17.821040766s) [upload=4749]: Got next segment. Inline: false Aug 7 21:02:04.234049276 (4h35m17.821105517s) [upload=4749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:02:04.234058734 (4h35m17.821114974s) [upload=4749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:02:04.237083389 (4h35m17.824139641s) [upload=4749]: Next segment err: metaclient: Too Many Requests Aug 7 21:02:04.237137491 (4h35m17.824193731s) [upload=4749]: Done uploading segments... Aug 7 21:02:04.237155915 (4h35m17.824212156s) [upload=4749]: Upload finished. err: metaclient: Too Many Requests Duration: 563.915331ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 5m41.035929323s FAIL (2025-08-07 21:17:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5m41.035953329s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 25.048899ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:17:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 25.065679ms edge gateway vhost us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)