Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.X..................................XXXXXXXXXXX] duration: 30.001376043s FAIL (2025-08-07 22:19:01 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001404596s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXX.XX.........................................XXXXXXXXX] duration: 30.00026303s FAIL (2025-08-07 22:18:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000280983s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX....................................XXXXXXX] duration: 1m0.001490814s FAIL (2025-08-07 22:18:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001503639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXX...X.....X..............................XXXXXXXXX] duration: 30.002111882s FAIL (2025-08-07 22:18:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002134313s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XX......X.............................XXXXXXX] duration: 1m0.000587983s FAIL (2025-08-07 22:18:22 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== DONB5V57E2AE5RG55CEAS6WPI6L7VWFQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.534357043 (5h43m14.121413254s) [noise=true , address=104.36.82.58:4279 , piece_id=Z2FXWDDE6Q4S5P73GMZFI6RM5LBBRWMDLKH6XK7MF3IAUWZC6PIA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.552361511 (5h43m14.139417732s) [noise=true , address=104.36.82.4:4827 , piece_id=QP6JUIBNRV7YFX433AN5PLRGTRPQQ2XC52HF7EYQSGF3FMAXM4JA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.596411237 (5h43m14.183467448s) [noise=true , address=104.36.82.56:4241 , piece_id=CZNEIWPUR5RWWSRD42ZZV76SKKYQUL22K5OSMRABZ3JLNWPOQQPQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.596432907 (5h43m14.183489128s) [seg_pos={0 0}, upload=5882]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:00.596673115 (5h43m14.183729336s) [noise=true , address=104.36.82.13:4197 , piece_id=YXUCKQYQYWVKLTOR6AZHGS5LANJQESFHAGE6F2ZA272CZDT2BUIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection; write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection Aug 7 22:10:00.596697370 (5h43m14.183753591s) [noise=true , address=104.36.82.16:4327 , piece_id=GRKTSALMEB73JLA74SJ6ET75F3JY4PXVTYOGTANJBS3VOETOVBIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection; write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection Aug 7 22:10:00.596712248 (5h43m14.183768469s) [noise=true , address=79.127.205.234:4600 , piece_id=LP74NCGX4H2E4U5UHJATTAARN6E4O4MVSVP2ED2N67LXJQX7DPAQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kxaNeRZqxza8AxSaF4PvBkmKoUY2NvficDEBRaXfDnq1HSgJ1): protocol: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection; write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection Aug 7 22:10:00.596771750 (5h43m14.183827971s) [noise=true , address=104.36.82.37:4685 , piece_id=4Q4WHVJWI4VXAEZEPTS355RHEBI37ME5S3MDUI5KM45LVDZ3GZ3A, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MhzeNvbKYhZVnhQW8oY5BeGQDuyNtmi25Xhgzrej4bCyoD9VM): protocol: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection Aug 7 22:10:00.596795714 (5h43m14.183851935s) [seg_pos={0 0}, upload=5882]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:00.596866456 (5h43m14.183922677s) [upload=5882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:00.596873089 (5h43m14.18392931s) [upload=5882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:00.660018838 (5h43m14.247075089s) [upload=5882]: Done uploading segments... Aug 7 22:10:00.660039907 (5h43m14.247096149s) [upload=5882]: Upload finished. err: <nil> Aug 7 22:11:14.838405386 (5h44m28.425461627s) [upload=5904]: Starting upload Aug 7 22:11:14.838502537 (5h44m28.425558788s) [upload=5904]: Done starting upload Aug 7 22:11:14.838673375 (5h44m28.425729627s) [upload=5904]: Uploading segments... Aug 7 22:11:14.838929564 (5h44m28.425985815s) [upload=5904]: Got next segment. Inline: false Aug 7 22:11:14.838997871 (5h44m28.426054112s) [upload=5904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:14.839003762 (5h44m28.426060003s) [upload=5904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:14.826003908 (5h45m28.413060119s) [upload=5904]: Next segment err: context deadline exceeded Aug 7 22:12:14.826061727 (5h45m28.413117948s) [upload=5904]: Done uploading segments... Aug 7 22:12:14.826077997 (5h45m28.413134218s) [upload=5904]: Upload finished. err: context deadline exceeded Aug 7 22:12:14.896180564 (5h45m28.483236775s) [upload=5918]: Starting upload Aug 7 22:12:14.896231629 (5h45m28.483287841s) [upload=5918]: Done starting upload Aug 7 22:12:14.896288415 (5h45m28.483344636s) [upload=5918]: Uploading segments... Aug 7 22:12:14.896472659 (5h45m28.48352888s) [upload=5918]: Got next segment. Inline: false Aug 7 22:12:14.896503706 (5h45m28.483559918s) [upload=5918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:14.896507904 (5h45m28.483564126s) [upload=5918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:14.884243492 (5h46m28.471299734s) [upload=5918]: Next segment err: context deadline exceeded Aug 7 22:13:14.884316208 (5h46m28.471372449s) [upload=5918]: Done uploading segments... Aug 7 22:13:14.884334913 (5h46m28.471391154s) [upload=5918]: Upload finished. err: context deadline exceeded Aug 7 22:13:15.003888637 (5h46m28.590944898s) [upload=5932]: Starting upload Aug 7 22:13:15.004039338 (5h46m28.591095609s) [upload=5932]: Done starting upload Aug 7 22:13:15.004748231 (5h46m28.591804483s) [upload=5932]: Uploading segments... Aug 7 22:13:15.004838500 (5h46m28.591894741s) [upload=5932]: Got next segment. Inline: false Aug 7 22:13:15.004895547 (5h46m28.591951788s) [upload=5932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:15.004902439 (5h46m28.591958681s) [upload=5932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:14.988056084 (5h47m28.575112325s) [upload=5932]: Next segment err: context deadline exceeded Aug 7 22:14:14.988128109 (5h47m28.575184351s) [upload=5932]: Done uploading segments... Aug 7 22:14:14.988155329 (5h47m28.57521157s) [upload=5932]: Upload finished. err: context deadline exceeded Aug 7 22:14:22.127816864 (5h47m35.714873075s) [upload=5953]: Starting upload Aug 7 22:14:22.127868040 (5h47m35.714924261s) [upload=5953]: Done starting upload Aug 7 22:14:22.127944151 (5h47m35.715000372s) [upload=5953]: Uploading segments... Aug 7 22:14:22.128112817 (5h47m35.715169038s) [upload=5953]: Got next segment. Inline: false Aug 7 22:14:22.128145027 (5h47m35.715201248s) [upload=5953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:22.128148994 (5h47m35.715205205s) [upload=5953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:22.111818521 (5h48m35.698874742s) [upload=5953]: Next segment err: context deadline exceeded Aug 7 22:15:22.111891768 (5h48m35.698947999s) [upload=5953]: Done uploading segments... Aug 7 22:15:22.111945839 (5h48m35.69900206s) [upload=5953]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.210077596 (5h48m35.797133807s) [upload=5967]: Starting upload Aug 7 22:15:22.210126076 (5h48m35.797182286s) [upload=5967]: Done starting upload Aug 7 22:15:22.210213909 (5h48m35.79727013s) [upload=5967]: Uploading segments... Aug 7 22:15:22.210350815 (5h48m35.797407026s) [upload=5967]: Got next segment. Inline: false Aug 7 22:15:22.210380320 (5h48m35.797436542s) [upload=5967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.210384398 (5h48m35.797440619s) [upload=5967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.196775688 (5h49m35.783831939s) [upload=5967]: Next segment err: context deadline exceeded Aug 7 22:16:22.196861187 (5h49m35.783917439s) [upload=5967]: Done uploading segments... Aug 7 22:16:22.196885322 (5h49m35.783941574s) [upload=5967]: Upload finished. err: context deadline exceeded Aug 7 22:16:22.269422973 (5h49m35.856479254s) [upload=5981]: Starting upload Aug 7 22:16:22.269535553 (5h49m35.856591794s) [upload=5981]: Done starting upload Aug 7 22:16:22.269676246 (5h49m35.856732487s) [upload=5981]: Uploading segments... Aug 7 22:16:22.269948395 (5h49m35.857004635s) [upload=5981]: Got next segment. Inline: false Aug 7 22:16:22.269993167 (5h49m35.857049419s) [upload=5981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:22.269999790 (5h49m35.857056041s) [upload=5981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:22.257287873 (5h50m35.844344113s) [upload=5981]: Next segment err: context deadline exceeded Aug 7 22:17:22.257515257 (5h50m35.844571508s) [upload=5981]: Done uploading segments... Aug 7 22:17:22.257574649 (5h50m35.844630889s) [upload=5981]: Upload finished. err: context deadline exceeded Aug 7 22:17:22.352562567 (5h50m35.939618808s) [upload=5994]: Starting upload Aug 7 22:17:22.352642467 (5h50m35.939698707s) [upload=5994]: Done starting upload Aug 7 22:17:22.352739408 (5h50m35.939795648s) [upload=5994]: Uploading segments... Aug 7 22:17:22.353012627 (5h50m35.940068868s) [upload=5994]: Got next segment. Inline: false Aug 7 22:17:22.353063573 (5h50m35.940119824s) [upload=5994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:22.353069433 (5h50m35.940125675s) [upload=5994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:22.334060789 (5h51m35.92111701s) [upload=5994]: Next segment err: context deadline exceeded Aug 7 22:18:22.334109780 (5h51m35.921165992s) [upload=5994]: Done uploading segments... Aug 7 22:18:22.334124208 (5h51m35.921180419s) [upload=5994]: Upload finished. err: context deadline exceeded Duration: 1m0.00060162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXX..X........................................XXXXXXXX] duration: 30.000356589s FAIL (2025-08-07 22:18:13 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 55]: Uploading piece... Aug 7 22:14:51.426300299 (5h48m5.013356541s) [noise=true , address=77.90.31.15:4697 , piece_id=XDCICPLIEUA3HBHPTL5YKQ5MITBASANALKMAXLEAATHSTA23HWBQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.426661613 (5h48m5.013717855s) [noise=true , address=45.91.116.10:4082 , piece_id=MCI5J5AEY44AJPHKEVSTVZGDBHRNMG4R5GCIASZ2OXX4F2JMIKMQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.427149574 (5h48m5.014205816s) [noise=true , address=77.90.31.17:4772 , piece_id=WLFZC5FRPMYYQDN4JZVPYDCNHDMVMNL2AKV4F5UKQQ7JM4JYXDFA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.427626695 (5h48m5.014682937s) [noise=true , address=77.90.31.16:4735 , piece_id=HQ2BPD52ZQXRH5YHMX7JGS2JJ5I5BQALG327UMWAYUW7W245N4QQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428103996 (5h48m5.015160247s) [noise=true , address=104.36.82.6:4873 , piece_id=D5QPVOWRUSHU5HITFOBHNYS3J7REL5XOROWGLJC4EAHXNH7EJ73A, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428503051 (5h48m5.015559292s) [noise=true , address=104.36.82.7:4912 , piece_id=QBFXRWNLJKWEOJ36KB6AJMOBZ64BW5NT4UZINBVGQHQZDLFXP5YA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428861140 (5h48m5.01591739s) [noise=true , address=45.91.116.5:4299 , piece_id=KP6S3WEZGKH2V5OE34LLHHSNQ5XPRKSKFNK66DVOTMWOWBJCOZQA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.429259873 (5h48m5.016316115s) [noise=true , address=45.91.116.9:4261 , piece_id=JNPPGNLOROFBT27FHNEU6HQY77NEPFQBZOHJZRDAPCUO3UOCKR2Q, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.429662896 (5h48m5.016719138s) [noise=true , address=77.90.31.17:4770 , piece_id=EGU57FLC2PJOV3AXAK2NDGYGHISFH3L3GFK5ZMWUM45MG3RUUOOA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.550946177 (5h48m5.138002409s) [upload=5955]: Next returned nil segment Aug 7 22:14:51.550961846 (5h48m5.138018078s) [upload=5955]: Waiting for error group managing segments... Aug 7 22:14:51.559757896 (5h48m5.146814117s) [noise=true , address=45.91.116.14:4189 , piece_id=65YS2EZ7TJFMO2RHD6F2SGUN2CXHJ47AI3NOX4G45GCKR663LXTQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561017878 (5h48m5.148074129s) [noise=true , address=45.91.116.12:4227 , piece_id=PT72AOCQEVQJ2ZSMYVM4FHBQSKMJDSGMKRAHRRS6W4QVTWNIYEAQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561713717 (5h48m5.148769938s) [noise=true , address=45.91.116.10:4082 , piece_id=MCI5J5AEY44AJPHKEVSTVZGDBHRNMG4R5GCIASZ2OXX4F2JMIKMQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561816609 (5h48m5.14887284s) [noise=true , address=45.91.116.5:4299 , piece_id=KP6S3WEZGKH2V5OE34LLHHSNQ5XPRKSKFNK66DVOTMWOWBJCOZQA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.562321571 (5h48m5.149377812s) [noise=true , address=45.91.116.9:4261 , piece_id=JNPPGNLOROFBT27FHNEU6HQY77NEPFQBZOHJZRDAPCUO3UOCKR2Q, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.563667614 (5h48m5.150723845s) [noise=true , address=77.90.31.15:4697 , piece_id=XDCICPLIEUA3HBHPTL5YKQ5MITBASANALKMAXLEAATHSTA23HWBQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.565443659 (5h48m5.15249991s) [noise=true , address=104.36.82.15:4257 , piece_id=C6V3O2TFBIF5W2E4AJA6BIRK7CJOQUJ73H5SVYPB5KYXUZ3HN5PA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.566941915 (5h48m5.153998147s) [noise=true , address=104.36.82.7:4912 , piece_id=QBFXRWNLJKWEOJ36KB6AJMOBZ64BW5NT4UZINBVGQHQZDLFXP5YA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.567819224 (5h48m5.154875466s) [noise=true , address=45.91.116.14:4190 , piece_id=D64JHW36T5LGBYG6IDZVYQD4GZWCYQI55LR4QOHOPCDRTUJJHMYA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.568503540 (5h48m5.155559782s) [noise=true , address=104.36.82.6:4873 , piece_id=D5QPVOWRUSHU5HITFOBHNYS3J7REL5XOROWGLJC4EAHXNH7EJ73A, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.576217691 (5h48m5.163273931s) [noise=true , address=45.91.116.12:4226 , piece_id=2YPECU45KNEFQEABDWQNUY6DTGZZWUFE5YK767DQSZKXW6AHJBDA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.578272647 (5h48m5.165328888s) [noise=true , address=77.90.31.17:4770 , piece_id=EGU57FLC2PJOV3AXAK2NDGYGHISFH3L3GFK5ZMWUM45MG3RUUOOA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.735239817 (5h48m5.322296068s) [noise=true , address=104.36.82.16:4294 , piece_id=GM6LYNFPDJVGPTJA7ZIQYZFEIMNSCQYRSNO4TPWYZJJ6S6N63HNA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.735279151 (5h48m5.322335392s) [seg_pos={0 0}, upload=5955]: Segment reached optimal threshold of 13 pieces. Aug 7 22:14:51.735633281 (5h48m5.322689532s) [noise=true , address=77.90.31.17:4772 , piece_id=WLFZC5FRPMYYQDN4JZVPYDCNHDMVMNL2AKV4F5UKQQ7JM4JYXDFA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection; write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection Aug 7 22:14:51.735644341 (5h48m5.322700583s) [noise=true , address=77.90.31.16:4735 , piece_id=HQ2BPD52ZQXRH5YHMX7JGS2JJ5I5BQALG327UMWAYUW7W245N4QQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection Aug 7 22:14:51.735698282 (5h48m5.322754524s) [noise=true , address=77.90.31.13:4626 , piece_id=4WWEMP7VRA72YTDSGUSQHVACDKAPJNH5JO3XJQQDS5A3VY7HLENA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection; write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection Aug 7 22:14:51.735750810 (5h48m5.322807052s) [seg_pos={0 0}, upload=5955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:14:51.735825741 (5h48m5.322881982s) [upload=5955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:14:51.735831010 (5h48m5.322887252s) [upload=5955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:28.158244845 (5h48m41.745301096s) [upload=5955]: Done uploading segments... Aug 7 22:15:28.158274540 (5h48m41.74533079s) [upload=5955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:28.250312080 (5h48m41.837368322s) [upload=5969]: Starting upload Aug 7 22:15:28.250412248 (5h48m41.837468498s) [upload=5969]: Done starting upload Aug 7 22:15:28.250539435 (5h48m41.837595676s) [upload=5969]: Uploading segments... Aug 7 22:15:28.250814920 (5h48m41.83787116s) [upload=5969]: Got next segment. Inline: false Aug 7 22:15:28.250863259 (5h48m41.8379195s) [upload=5969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:28.250871685 (5h48m41.837927936s) [upload=5969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:58.237095607 (5h49m11.824151818s) [upload=5969]: Next segment err: context deadline exceeded Aug 7 22:15:58.237308174 (5h49m11.824364395s) [upload=5969]: Done uploading segments... Aug 7 22:15:58.237357947 (5h49m11.824414157s) [upload=5969]: Upload finished. err: context deadline exceeded Aug 7 22:16:42.428984950 (5h49m56.016041171s) [upload=5985]: Starting upload Aug 7 22:16:42.429041335 (5h49m56.016097556s) [upload=5985]: Done starting upload Aug 7 22:16:42.429149126 (5h49m56.016205347s) [upload=5985]: Uploading segments... Aug 7 22:16:42.429280742 (5h49m56.016336963s) [upload=5985]: Got next segment. Inline: false Aug 7 22:16:42.429312962 (5h49m56.016369183s) [upload=5985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:42.429316729 (5h49m56.01637295s) [upload=5985]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:12.410858597 (5h50m25.997914808s) [upload=5985]: Next segment err: context deadline exceeded Aug 7 22:17:12.411035658 (5h50m25.998091879s) [upload=5985]: Done uploading segments... Aug 7 22:17:12.411065273 (5h50m25.998121494s) [upload=5985]: Upload finished. err: context deadline exceeded Aug 7 22:17:43.138064432 (5h50m56.725120653s) [upload=6000]: Starting upload Aug 7 22:17:43.138125136 (5h50m56.725181357s) [upload=6000]: Done starting upload Aug 7 22:17:43.138238818 (5h50m56.725295039s) [upload=6000]: Uploading segments... Aug 7 22:17:43.138379291 (5h50m56.725435501s) [upload=6000]: Got next segment. Inline: false Aug 7 22:17:43.138408124 (5h50m56.725464335s) [upload=6000]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:43.138412112 (5h50m56.725468333s) [upload=6000]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:13.124828097 (5h51m26.711884319s) [upload=6000]: Next segment err: context deadline exceeded Aug 7 22:18:13.124916743 (5h51m26.711972954s) [upload=6000]: Done uploading segments... Aug 7 22:18:13.124930328 (5h51m26.711986539s) [upload=6000]: Upload finished. err: context deadline exceeded Duration: 30.000373742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X..................................XXXXXXXXXX] duration: 30.000889554s FAIL (2025-08-07 22:18:12 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000911626s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXX.............................XXXXXXXXXX] duration: 30.000464561s FAIL (2025-08-07 22:18:11 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : <nil> Aug 7 22:11:10.710381989 (5h44m24.29743823s) [upload=5902]: Starting upload Aug 7 22:11:10.710469673 (5h44m24.297525913s) [upload=5902]: Done starting upload Aug 7 22:11:10.710537730 (5h44m24.297593971s) [upload=5902]: Uploading segments... Aug 7 22:11:10.710600456 (5h44m24.297656698s) [upload=5902]: Got next segment. Inline: true Aug 7 22:11:10.710607830 (5h44m24.297664072s) [upload=5902]: Waiting for error group managing segments... Aug 7 22:11:10.710658876 (5h44m24.297715118s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:10.710666100 (5h44m24.297722341s) [upload=5902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:10.710670057 (5h44m24.297726299s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:40.695011486 (5h44m54.282067737s) [upload=5902]: Done uploading segments... Aug 7 22:11:40.695031373 (5h44m54.282087624s) [upload=5902]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:11:40.801935294 (5h44m54.388991515s) [upload=5908]: Starting upload Aug 7 22:11:40.802009452 (5h44m54.389065673s) [upload=5908]: Done starting upload Aug 7 22:11:40.802061539 (5h44m54.38911776s) [upload=5908]: Uploading segments... Aug 7 22:11:40.802108137 (5h44m54.389164358s) [upload=5908]: Got next segment. Inline: true Aug 7 22:11:40.802112685 (5h44m54.389168896s) [upload=5908]: Waiting for error group managing segments... Aug 7 22:11:40.802146128 (5h44m54.389202338s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.802150235 (5h44m54.389206456s) [upload=5908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:40.802172947 (5h44m54.389229168s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.785746890 (5h45m24.372803111s) [upload=5908]: Done uploading segments... Aug 7 22:12:10.785765736 (5h45m24.372821946s) [upload=5908]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:34.227606352 (5h45m47.814662563s) [upload=5920]: Starting upload Aug 7 22:12:34.227679809 (5h45m47.81473602s) [upload=5920]: Done starting upload Aug 7 22:12:34.227726427 (5h45m47.814782637s) [upload=5920]: Uploading segments... Aug 7 22:12:34.227755791 (5h45m47.814812012s) [upload=5920]: Got next segment. Inline: true Aug 7 22:12:34.227759368 (5h45m47.814815579s) [upload=5920]: Waiting for error group managing segments... Aug 7 22:12:34.227789123 (5h45m47.814845344s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:34.227792860 (5h45m47.814849081s) [upload=5920]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:34.227795325 (5h45m47.814851546s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:04.214429951 (5h46m17.801486172s) [upload=5920]: Done uploading segments... Aug 7 22:13:04.214444368 (5h46m17.801500579s) [upload=5920]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:21.693003516 (5h46m35.280059727s) [upload=5934]: Starting upload Aug 7 22:13:21.693054191 (5h46m35.280110402s) [upload=5934]: Done starting upload Aug 7 22:13:21.693097442 (5h46m35.280153663s) [upload=5934]: Uploading segments... Aug 7 22:13:21.693124161 (5h46m35.280180382s) [upload=5934]: Got next segment. Inline: true Aug 7 22:13:21.693127627 (5h46m35.280183848s) [upload=5934]: Waiting for error group managing segments... Aug 7 22:13:21.693172341 (5h46m35.280228552s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:21.693176398 (5h46m35.28023262s) [upload=5934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:21.693178873 (5h46m35.280235085s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:51.677769243 (5h47m5.264825484s) [upload=5934]: Done uploading segments... Aug 7 22:13:51.677786175 (5h47m5.264842416s) [upload=5934]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:51.763566605 (5h47m5.350622826s) [upload=5942]: Starting upload Aug 7 22:13:51.763615116 (5h47m5.350671337s) [upload=5942]: Done starting upload Aug 7 22:13:51.763654910 (5h47m5.350711131s) [upload=5942]: Uploading segments... Aug 7 22:13:51.763683413 (5h47m5.350739634s) [upload=5942]: Got next segment. Inline: true Aug 7 22:13:51.763687000 (5h47m5.350743211s) [upload=5942]: Waiting for error group managing segments... Aug 7 22:13:51.763718620 (5h47m5.350774841s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.763722296 (5h47m5.350778517s) [upload=5942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:51.763724580 (5h47m5.350780801s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:21.752690510 (5h47m35.339746731s) [upload=5942]: Done uploading segments... Aug 7 22:14:21.752712410 (5h47m35.339768631s) [upload=5942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:21.832761353 (5h47m35.419817564s) [upload=5952]: Starting upload Aug 7 22:14:21.832808701 (5h47m35.419864913s) [upload=5952]: Done starting upload Aug 7 22:14:21.832873814 (5h47m35.419930035s) [upload=5952]: Uploading segments... Aug 7 22:14:21.832905782 (5h47m35.419961993s) [upload=5952]: Got next segment. Inline: true Aug 7 22:14:21.832909440 (5h47m35.419965661s) [upload=5952]: Waiting for error group managing segments... Aug 7 22:14:21.832940607 (5h47m35.419996828s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:21.832944185 (5h47m35.420000406s) [upload=5952]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:21.832946779 (5h47m35.420003s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:51.821960613 (5h48m5.409016855s) [upload=5952]: Done uploading segments... Aug 7 22:14:51.821995769 (5h48m5.40905202s) [upload=5952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:07.153250299 (5h48m20.74030654s) [upload=5963]: Starting upload Aug 7 22:15:07.153363971 (5h48m20.740420213s) [upload=5963]: Done starting upload Aug 7 22:15:07.153422571 (5h48m20.740478811s) [upload=5963]: Uploading segments... Aug 7 22:15:07.153473596 (5h48m20.740529837s) [upload=5963]: Got next segment. Inline: true Aug 7 22:15:07.153480569 (5h48m20.74053681s) [upload=5963]: Waiting for error group managing segments... Aug 7 22:15:07.153529239 (5h48m20.740585481s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:07.153535742 (5h48m20.740591973s) [upload=5963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:07.153539228 (5h48m20.740595469s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:37.139229671 (5h48m50.726285892s) [upload=5963]: Done uploading segments... Aug 7 22:15:37.139243938 (5h48m50.726300148s) [upload=5963]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:37.225059294 (5h48m50.812115505s) [upload=5970]: Starting upload Aug 7 22:15:37.225107253 (5h48m50.812163474s) [upload=5970]: Done starting upload Aug 7 22:15:37.225145674 (5h48m50.812201895s) [upload=5970]: Uploading segments... Aug 7 22:15:37.225193384 (5h48m50.812249605s) [upload=5970]: Got next segment. Inline: true Aug 7 22:15:37.225196900 (5h48m50.81225311s) [upload=5970]: Waiting for error group managing segments... Aug 7 22:15:37.225226886 (5h48m50.812283107s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:37.225231044 (5h48m50.812287254s) [upload=5970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:37.225233308 (5h48m50.812289529s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:07.214043839 (5h49m20.80110009s) [upload=5970]: Done uploading segments... Aug 7 22:16:07.214071880 (5h49m20.801128131s) [upload=5970]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:27.239225384 (5h49m40.826281605s) [upload=5983]: Starting upload Aug 7 22:16:27.239274204 (5h49m40.826330425s) [upload=5983]: Done starting upload Aug 7 22:16:27.239318858 (5h49m40.826375079s) [upload=5983]: Uploading segments... Aug 7 22:16:27.239353452 (5h49m40.826409673s) [upload=5983]: Got next segment. Inline: true Aug 7 22:16:27.239358842 (5h49m40.826415063s) [upload=5983]: Waiting for error group managing segments... Aug 7 22:16:27.239399728 (5h49m40.82645595s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:27.239405820 (5h49m40.826462041s) [upload=5983]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:27.239410048 (5h49m40.826466269s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:57.228025199 (5h50m10.81508144s) [upload=5983]: Done uploading segments... Aug 7 22:16:57.228050357 (5h50m10.815106607s) [upload=5983]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:41.748643128 (5h50m55.33569938s) [upload=5998]: Starting upload Aug 7 22:17:41.748800843 (5h50m55.335857084s) [upload=5998]: Done starting upload Aug 7 22:17:41.748866916 (5h50m55.335923157s) [upload=5998]: Uploading segments... Aug 7 22:17:41.748948899 (5h50m55.33600514s) [upload=5998]: Got next segment. Inline: true Aug 7 22:17:41.748956914 (5h50m55.336013155s) [upload=5998]: Waiting for error group managing segments... Aug 7 22:17:41.749006376 (5h50m55.336062618s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:41.749012397 (5h50m55.336068639s) [upload=5998]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:41.749016225 (5h50m55.336072466s) [upload=5998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:11.736773915 (5h51m25.323830136s) [upload=5998]: Done uploading segments... Aug 7 22:18:11.736788051 (5h51m25.323844272s) [upload=5998]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000482874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 193.465708ms FAIL (2025-08-07 22:18:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 193.479193ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.X.XX...............................XXXXX] duration: 1m0.000573892s FAIL (2025-08-07 22:18:04 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m10.237669272s) [upload=5826]: Done uploading segments... Aug 7 22:06:56.650627668 (5h40m10.237683889s) [upload=5826]: Upload finished. err: <nil> Aug 7 22:07:58.192526348 (5h41m11.77958259s) [upload=5845]: Starting upload Aug 7 22:07:58.192614623 (5h41m11.779670864s) [upload=5845]: Done starting upload Aug 7 22:07:58.192700764 (5h41m11.779757014s) [upload=5845]: Uploading segments... Aug 7 22:07:58.192744676 (5h41m11.779800917s) [upload=5845]: Got next segment. Inline: true Aug 7 22:07:58.192750096 (5h41m11.779806337s) [upload=5845]: Waiting for error group managing segments... Aug 7 22:07:58.192795772 (5h41m11.779852013s) [upload=5845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:58.192801672 (5h41m11.779857913s) [upload=5845]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:58.192805169 (5h41m11.779861409s) [upload=5845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:58.264015693 (5h41m11.851071934s) [upload=5845]: Done uploading segments... Aug 7 22:07:58.264038786 (5h41m11.851095027s) [upload=5845]: Upload finished. err: <nil> Aug 7 22:09:12.971535319 (5h42m26.558591531s) [upload=5870]: Starting upload Aug 7 22:09:12.971593908 (5h42m26.558650129s) [upload=5870]: Done starting upload Aug 7 22:09:12.971640085 (5h42m26.558696306s) [upload=5870]: Uploading segments... Aug 7 22:09:12.971667336 (5h42m26.558723547s) [upload=5870]: Got next segment. Inline: true Aug 7 22:09:12.971670973 (5h42m26.558727184s) [upload=5870]: Waiting for error group managing segments... Aug 7 22:09:12.971701830 (5h42m26.558758051s) [upload=5870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:12.971705557 (5h42m26.558761768s) [upload=5870]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:12.971707941 (5h42m26.558764152s) [upload=5870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:13.026541326 (5h42m26.613597568s) [upload=5870]: Done uploading segments... Aug 7 22:09:13.026558519 (5h42m26.61361477s) [upload=5870]: Upload finished. err: <nil> Aug 7 22:10:14.601261519 (5h43m28.188317731s) [upload=5887]: Starting upload Aug 7 22:10:14.601309920 (5h43m28.188366141s) [upload=5887]: Done starting upload Aug 7 22:10:14.601350215 (5h43m28.188406436s) [upload=5887]: Uploading segments... Aug 7 22:10:14.601378688 (5h43m28.188434909s) [upload=5887]: Got next segment. Inline: true Aug 7 22:10:14.601382325 (5h43m28.188438546s) [upload=5887]: Waiting for error group managing segments... Aug 7 22:10:14.601411149 (5h43m28.188467361s) [upload=5887]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:14.601414947 (5h43m28.188471168s) [upload=5887]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:14.601417170 (5h43m28.188473391s) [upload=5887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:14.743486813 (5h43m28.330543034s) [upload=5887]: Done uploading segments... Aug 7 22:10:14.743500428 (5h43m28.33055664s) [upload=5887]: Upload finished. err: <nil> Aug 7 22:11:00.429392635 (5h44m14.016448856s) [upload=5899]: Starting upload Aug 7 22:11:00.429436647 (5h44m14.016492868s) [upload=5899]: Done starting upload Aug 7 22:11:00.429474658 (5h44m14.016530879s) [upload=5899]: Uploading segments... Aug 7 22:11:00.429501167 (5h44m14.016557378s) [upload=5899]: Got next segment. Inline: true Aug 7 22:11:00.429504493 (5h44m14.016560714s) [upload=5899]: Waiting for error group managing segments... Aug 7 22:11:00.429533688 (5h44m14.016589909s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:00.429537505 (5h44m14.016593717s) [upload=5899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:00.429540060 (5h44m14.016596281s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:00.493866886 (5h44m14.080923097s) [upload=5899]: Done uploading segments... Aug 7 22:11:00.493879859 (5h44m14.08093608s) [upload=5899]: Upload finished. err: <nil> Aug 7 22:12:13.235953272 (5h45m26.823009493s) [upload=5915]: Starting upload Aug 7 22:12:13.236008135 (5h45m26.823064356s) [upload=5915]: Done starting upload Aug 7 22:12:13.236053961 (5h45m26.823110172s) [upload=5915]: Uploading segments... Aug 7 22:12:13.236083775 (5h45m26.823139997s) [upload=5915]: Got next segment. Inline: true Aug 7 22:12:13.236087523 (5h45m26.823143733s) [upload=5915]: Waiting for error group managing segments... Aug 7 22:12:13.236120344 (5h45m26.823176555s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:13.236123931 (5h45m26.823180142s) [upload=5915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:13.236126285 (5h45m26.823182506s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:13.466422283 (5h45m27.053478504s) [upload=5915]: Done uploading segments... Aug 7 22:12:13.466437992 (5h45m27.053494213s) [upload=5915]: Upload finished. err: <nil> Aug 7 22:12:59.930261536 (5h46m13.517317778s) [upload=5926]: Starting upload Aug 7 22:12:59.930344330 (5h46m13.517400582s) [upload=5926]: Done starting upload Aug 7 22:12:59.930411596 (5h46m13.517467837s) [upload=5926]: Uploading segments... Aug 7 22:12:59.930461800 (5h46m13.517518041s) [upload=5926]: Got next segment. Inline: true Aug 7 22:12:59.930467751 (5h46m13.517523992s) [upload=5926]: Waiting for error group managing segments... Aug 7 22:12:59.930514368 (5h46m13.51757061s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:59.930520589 (5h46m13.51757683s) [upload=5926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:59.930524356 (5h46m13.517580607s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:59.913184211 (5h47m13.500240423s) [upload=5926]: Done uploading segments... Aug 7 22:13:59.913207345 (5h47m13.500263566s) [upload=5926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:00.010650606 (5h47m13.597706828s) [upload=5944]: Starting upload Aug 7 22:14:00.010701831 (5h47m13.597758043s) [upload=5944]: Done starting upload Aug 7 22:14:00.010753608 (5h47m13.59780983s) [upload=5944]: Uploading segments... Aug 7 22:14:00.010809803 (5h47m13.597866024s) [upload=5944]: Got next segment. Inline: true Aug 7 22:14:00.010813610 (5h47m13.597869821s) [upload=5944]: Waiting for error group managing segments... Aug 7 22:14:00.010845560 (5h47m13.597901781s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:00.010849677 (5h47m13.597905899s) [upload=5944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:00.010851892 (5h47m13.597908113s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:59.990902753 (5h48m13.577958974s) [upload=5944]: Done uploading segments... Aug 7 22:14:59.990918592 (5h48m13.577974803s) [upload=5944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:04.091713143 (5h48m17.678769415s) [upload=5961]: Starting upload Aug 7 22:15:04.091810745 (5h48m17.678866986s) [upload=5961]: Done starting upload Aug 7 22:15:04.091875977 (5h48m17.678932217s) [upload=5961]: Uploading segments... Aug 7 22:15:04.091965724 (5h48m17.679021975s) [upload=5961]: Got next segment. Inline: true Aug 7 22:15:04.091976223 (5h48m17.679032475s) [upload=5961]: Waiting for error group managing segments... Aug 7 22:15:04.092046285 (5h48m17.679102526s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:04.092055141 (5h48m17.679111383s) [upload=5961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:04.092061392 (5h48m17.679117633s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.079406942 (5h49m17.666463153s) [upload=5961]: Done uploading segments... Aug 7 22:16:04.079423422 (5h49m17.666479644s) [upload=5961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:04.164640982 (5h49m17.751697203s) [upload=5974]: Starting upload Aug 7 22:16:04.164694422 (5h49m17.751750633s) [upload=5974]: Done starting upload Aug 7 22:16:04.164734035 (5h49m17.751790246s) [upload=5974]: Uploading segments... Aug 7 22:16:04.164772067 (5h49m17.751828288s) [upload=5974]: Got next segment. Inline: true Aug 7 22:16:04.164776555 (5h49m17.751832776s) [upload=5974]: Waiting for error group managing segments... Aug 7 22:16:04.164825836 (5h49m17.751882047s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:04.164830154 (5h49m17.751886375s) [upload=5974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:04.164833621 (5h49m17.751889842s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:04.148146721 (5h50m17.735202963s) [upload=5974]: Done uploading segments... Aug 7 22:17:04.148198359 (5h50m17.735254599s) [upload=5974]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:04.228585182 (5h50m17.815641403s) [upload=5990]: Starting upload Aug 7 22:17:04.228636998 (5h50m17.815693219s) [upload=5990]: Done starting upload Aug 7 22:17:04.228690769 (5h50m17.81574699s) [upload=5990]: Uploading segments... Aug 7 22:17:04.228721386 (5h50m17.815777607s) [upload=5990]: Got next segment. Inline: true Aug 7 22:17:04.228725082 (5h50m17.815781303s) [upload=5990]: Waiting for error group managing segments... Aug 7 22:17:04.228765939 (5h50m17.815822161s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:04.228772071 (5h50m17.815828292s) [upload=5990]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:04.228775867 (5h50m17.815832089s) [upload=5990]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:04.215607888 (5h51m17.802664109s) [upload=5990]: Done uploading segments... Aug 7 22:18:04.215625421 (5h51m17.802681632s) [upload=5990]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000584962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXX.XX.........................................XXXXXXXX] duration: 30.000958679s FAIL (2025-08-07 22:18:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001070508s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 2m0.001236021s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:17:54 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.001257281s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX....................................XXXXXX] duration: 1m0.000503518s FAIL (2025-08-07 22:17:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000519958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXX.XX.........................................XXXXXXX] duration: 30.000643879s FAIL (2025-08-07 22:17:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00066573s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXX...X.....X..............................XXXXXXXX] duration: 30.001034137s FAIL (2025-08-07 22:17:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001049857s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XX......X.............................XXXXXX] duration: 1m0.001085007s FAIL (2025-08-07 22:17:22 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ET5VD3RYRGJWJK4PDGXG45IVJKB7SOSVP7YZA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.399754149 (5h43m13.98681037s) [noise=true , address=104.36.82.9:4011 , piece_id=UT6I3AJEWBCSPWUJPGJK3BUB6SZMFAZ5EQBAQ56Q65BFAAIIROIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.460854191 (5h43m14.047910432s) [noise=true , address=104.36.82.42:4799 , piece_id=LXQBRMRAEKV5OY2L5WGUW4E43RMLFBHYZRTZ2IUGMUDENRDBQLFQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.470345858 (5h43m14.0574021s) [noise=true , address=104.36.82.43:4846 , piece_id=3HN5X5L6FZOM2LY6UG6FUPFQSJGVCISOHFDI4T6IIW4XRCV7EKCQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.507622690 (5h43m14.094678901s) [noise=true , address=104.36.82.45:4913 , piece_id=FN5X5ZS7L7HRERGTFSBHDONB5V57E2AE5RG55CEAS6WPI6L7VWFQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.534357043 (5h43m14.121413254s) [noise=true , address=104.36.82.58:4279 , piece_id=Z2FXWDDE6Q4S5P73GMZFI6RM5LBBRWMDLKH6XK7MF3IAUWZC6PIA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.552361511 (5h43m14.139417732s) [noise=true , address=104.36.82.4:4827 , piece_id=QP6JUIBNRV7YFX433AN5PLRGTRPQQ2XC52HF7EYQSGF3FMAXM4JA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.596411237 (5h43m14.183467448s) [noise=true , address=104.36.82.56:4241 , piece_id=CZNEIWPUR5RWWSRD42ZZV76SKKYQUL22K5OSMRABZ3JLNWPOQQPQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.596432907 (5h43m14.183489128s) [seg_pos={0 0}, upload=5882]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:00.596673115 (5h43m14.183729336s) [noise=true , address=104.36.82.13:4197 , piece_id=YXUCKQYQYWVKLTOR6AZHGS5LANJQESFHAGE6F2ZA272CZDT2BUIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection; write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection Aug 7 22:10:00.596697370 (5h43m14.183753591s) [noise=true , address=104.36.82.16:4327 , piece_id=GRKTSALMEB73JLA74SJ6ET75F3JY4PXVTYOGTANJBS3VOETOVBIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection; write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection Aug 7 22:10:00.596712248 (5h43m14.183768469s) [noise=true , address=79.127.205.234:4600 , piece_id=LP74NCGX4H2E4U5UHJATTAARN6E4O4MVSVP2ED2N67LXJQX7DPAQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kxaNeRZqxza8AxSaF4PvBkmKoUY2NvficDEBRaXfDnq1HSgJ1): protocol: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection; write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection Aug 7 22:10:00.596771750 (5h43m14.183827971s) [noise=true , address=104.36.82.37:4685 , piece_id=4Q4WHVJWI4VXAEZEPTS355RHEBI37ME5S3MDUI5KM45LVDZ3GZ3A, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MhzeNvbKYhZVnhQW8oY5BeGQDuyNtmi25Xhgzrej4bCyoD9VM): protocol: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection Aug 7 22:10:00.596795714 (5h43m14.183851935s) [seg_pos={0 0}, upload=5882]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:00.596866456 (5h43m14.183922677s) [upload=5882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:00.596873089 (5h43m14.18392931s) [upload=5882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:00.660018838 (5h43m14.247075089s) [upload=5882]: Done uploading segments... Aug 7 22:10:00.660039907 (5h43m14.247096149s) [upload=5882]: Upload finished. err: <nil> Aug 7 22:11:14.838405386 (5h44m28.425461627s) [upload=5904]: Starting upload Aug 7 22:11:14.838502537 (5h44m28.425558788s) [upload=5904]: Done starting upload Aug 7 22:11:14.838673375 (5h44m28.425729627s) [upload=5904]: Uploading segments... Aug 7 22:11:14.838929564 (5h44m28.425985815s) [upload=5904]: Got next segment. Inline: false Aug 7 22:11:14.838997871 (5h44m28.426054112s) [upload=5904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:14.839003762 (5h44m28.426060003s) [upload=5904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:14.826003908 (5h45m28.413060119s) [upload=5904]: Next segment err: context deadline exceeded Aug 7 22:12:14.826061727 (5h45m28.413117948s) [upload=5904]: Done uploading segments... Aug 7 22:12:14.826077997 (5h45m28.413134218s) [upload=5904]: Upload finished. err: context deadline exceeded Aug 7 22:12:14.896180564 (5h45m28.483236775s) [upload=5918]: Starting upload Aug 7 22:12:14.896231629 (5h45m28.483287841s) [upload=5918]: Done starting upload Aug 7 22:12:14.896288415 (5h45m28.483344636s) [upload=5918]: Uploading segments... Aug 7 22:12:14.896472659 (5h45m28.48352888s) [upload=5918]: Got next segment. Inline: false Aug 7 22:12:14.896503706 (5h45m28.483559918s) [upload=5918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:14.896507904 (5h45m28.483564126s) [upload=5918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:14.884243492 (5h46m28.471299734s) [upload=5918]: Next segment err: context deadline exceeded Aug 7 22:13:14.884316208 (5h46m28.471372449s) [upload=5918]: Done uploading segments... Aug 7 22:13:14.884334913 (5h46m28.471391154s) [upload=5918]: Upload finished. err: context deadline exceeded Aug 7 22:13:15.003888637 (5h46m28.590944898s) [upload=5932]: Starting upload Aug 7 22:13:15.004039338 (5h46m28.591095609s) [upload=5932]: Done starting upload Aug 7 22:13:15.004748231 (5h46m28.591804483s) [upload=5932]: Uploading segments... Aug 7 22:13:15.004838500 (5h46m28.591894741s) [upload=5932]: Got next segment. Inline: false Aug 7 22:13:15.004895547 (5h46m28.591951788s) [upload=5932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:15.004902439 (5h46m28.591958681s) [upload=5932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:14.988056084 (5h47m28.575112325s) [upload=5932]: Next segment err: context deadline exceeded Aug 7 22:14:14.988128109 (5h47m28.575184351s) [upload=5932]: Done uploading segments... Aug 7 22:14:14.988155329 (5h47m28.57521157s) [upload=5932]: Upload finished. err: context deadline exceeded Aug 7 22:14:22.127816864 (5h47m35.714873075s) [upload=5953]: Starting upload Aug 7 22:14:22.127868040 (5h47m35.714924261s) [upload=5953]: Done starting upload Aug 7 22:14:22.127944151 (5h47m35.715000372s) [upload=5953]: Uploading segments... Aug 7 22:14:22.128112817 (5h47m35.715169038s) [upload=5953]: Got next segment. Inline: false Aug 7 22:14:22.128145027 (5h47m35.715201248s) [upload=5953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:22.128148994 (5h47m35.715205205s) [upload=5953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:22.111818521 (5h48m35.698874742s) [upload=5953]: Next segment err: context deadline exceeded Aug 7 22:15:22.111891768 (5h48m35.698947999s) [upload=5953]: Done uploading segments... Aug 7 22:15:22.111945839 (5h48m35.69900206s) [upload=5953]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.210077596 (5h48m35.797133807s) [upload=5967]: Starting upload Aug 7 22:15:22.210126076 (5h48m35.797182286s) [upload=5967]: Done starting upload Aug 7 22:15:22.210213909 (5h48m35.79727013s) [upload=5967]: Uploading segments... Aug 7 22:15:22.210350815 (5h48m35.797407026s) [upload=5967]: Got next segment. Inline: false Aug 7 22:15:22.210380320 (5h48m35.797436542s) [upload=5967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.210384398 (5h48m35.797440619s) [upload=5967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.196775688 (5h49m35.783831939s) [upload=5967]: Next segment err: context deadline exceeded Aug 7 22:16:22.196861187 (5h49m35.783917439s) [upload=5967]: Done uploading segments... Aug 7 22:16:22.196885322 (5h49m35.783941574s) [upload=5967]: Upload finished. err: context deadline exceeded Aug 7 22:16:22.269422973 (5h49m35.856479254s) [upload=5981]: Starting upload Aug 7 22:16:22.269535553 (5h49m35.856591794s) [upload=5981]: Done starting upload Aug 7 22:16:22.269676246 (5h49m35.856732487s) [upload=5981]: Uploading segments... Aug 7 22:16:22.269948395 (5h49m35.857004635s) [upload=5981]: Got next segment. Inline: false Aug 7 22:16:22.269993167 (5h49m35.857049419s) [upload=5981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:22.269999790 (5h49m35.857056041s) [upload=5981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:22.257287873 (5h50m35.844344113s) [upload=5981]: Next segment err: context deadline exceeded Aug 7 22:17:22.257515257 (5h50m35.844571508s) [upload=5981]: Done uploading segments... Aug 7 22:17:22.257574649 (5h50m35.844630889s) [upload=5981]: Upload finished. err: context deadline exceeded Duration: 1m0.001099934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.X..................................XXXXXXXXX] duration: 30.001761978s FAIL (2025-08-07 22:17:18 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001787437s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXX..X........................................XXXXXXX] duration: 30.00050705s FAIL (2025-08-07 22:17:12 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== EABDWQNUY6DTGZZWUFE5YK767DQSZKXW6AHJBDA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.424976539 (5h48m5.01203278s) [noise=true , address=45.91.116.12:4227 , piece_id=PT72AOCQEVQJ2ZSMYVM4FHBQSKMJDSGMKRAHRRS6W4QVTWNIYEAQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425100399 (5h48m5.01215661s) [noise=true , address=77.90.31.13:4626 , piece_id=4WWEMP7VRA72YTDSGUSQHVACDKAPJNH5JO3XJQQDS5A3VY7HLENA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425376945 (5h48m5.012433187s) [noise=true , address=104.36.82.16:4294 , piece_id=GM6LYNFPDJVGPTJA7ZIQYZFEIMNSCQYRSNO4TPWYZJJ6S6N63HNA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425848076 (5h48m5.012904316s) [noise=true , address=45.91.116.14:4190 , piece_id=D64JHW36T5LGBYG6IDZVYQD4GZWCYQI55LR4QOHOPCDRTUJJHMYA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.426300299 (5h48m5.013356541s) [noise=true , address=77.90.31.15:4697 , piece_id=XDCICPLIEUA3HBHPTL5YKQ5MITBASANALKMAXLEAATHSTA23HWBQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.426661613 (5h48m5.013717855s) [noise=true , address=45.91.116.10:4082 , piece_id=MCI5J5AEY44AJPHKEVSTVZGDBHRNMG4R5GCIASZ2OXX4F2JMIKMQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.427149574 (5h48m5.014205816s) [noise=true , address=77.90.31.17:4772 , piece_id=WLFZC5FRPMYYQDN4JZVPYDCNHDMVMNL2AKV4F5UKQQ7JM4JYXDFA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.427626695 (5h48m5.014682937s) [noise=true , address=77.90.31.16:4735 , piece_id=HQ2BPD52ZQXRH5YHMX7JGS2JJ5I5BQALG327UMWAYUW7W245N4QQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428103996 (5h48m5.015160247s) [noise=true , address=104.36.82.6:4873 , piece_id=D5QPVOWRUSHU5HITFOBHNYS3J7REL5XOROWGLJC4EAHXNH7EJ73A, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428503051 (5h48m5.015559292s) [noise=true , address=104.36.82.7:4912 , piece_id=QBFXRWNLJKWEOJ36KB6AJMOBZ64BW5NT4UZINBVGQHQZDLFXP5YA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428861140 (5h48m5.01591739s) [noise=true , address=45.91.116.5:4299 , piece_id=KP6S3WEZGKH2V5OE34LLHHSNQ5XPRKSKFNK66DVOTMWOWBJCOZQA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.429259873 (5h48m5.016316115s) [noise=true , address=45.91.116.9:4261 , piece_id=JNPPGNLOROFBT27FHNEU6HQY77NEPFQBZOHJZRDAPCUO3UOCKR2Q, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.429662896 (5h48m5.016719138s) [noise=true , address=77.90.31.17:4770 , piece_id=EGU57FLC2PJOV3AXAK2NDGYGHISFH3L3GFK5ZMWUM45MG3RUUOOA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.550946177 (5h48m5.138002409s) [upload=5955]: Next returned nil segment Aug 7 22:14:51.550961846 (5h48m5.138018078s) [upload=5955]: Waiting for error group managing segments... Aug 7 22:14:51.559757896 (5h48m5.146814117s) [noise=true , address=45.91.116.14:4189 , piece_id=65YS2EZ7TJFMO2RHD6F2SGUN2CXHJ47AI3NOX4G45GCKR663LXTQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561017878 (5h48m5.148074129s) [noise=true , address=45.91.116.12:4227 , piece_id=PT72AOCQEVQJ2ZSMYVM4FHBQSKMJDSGMKRAHRRS6W4QVTWNIYEAQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561713717 (5h48m5.148769938s) [noise=true , address=45.91.116.10:4082 , piece_id=MCI5J5AEY44AJPHKEVSTVZGDBHRNMG4R5GCIASZ2OXX4F2JMIKMQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561816609 (5h48m5.14887284s) [noise=true , address=45.91.116.5:4299 , piece_id=KP6S3WEZGKH2V5OE34LLHHSNQ5XPRKSKFNK66DVOTMWOWBJCOZQA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.562321571 (5h48m5.149377812s) [noise=true , address=45.91.116.9:4261 , piece_id=JNPPGNLOROFBT27FHNEU6HQY77NEPFQBZOHJZRDAPCUO3UOCKR2Q, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.563667614 (5h48m5.150723845s) [noise=true , address=77.90.31.15:4697 , piece_id=XDCICPLIEUA3HBHPTL5YKQ5MITBASANALKMAXLEAATHSTA23HWBQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.565443659 (5h48m5.15249991s) [noise=true , address=104.36.82.15:4257 , piece_id=C6V3O2TFBIF5W2E4AJA6BIRK7CJOQUJ73H5SVYPB5KYXUZ3HN5PA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.566941915 (5h48m5.153998147s) [noise=true , address=104.36.82.7:4912 , piece_id=QBFXRWNLJKWEOJ36KB6AJMOBZ64BW5NT4UZINBVGQHQZDLFXP5YA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.567819224 (5h48m5.154875466s) [noise=true , address=45.91.116.14:4190 , piece_id=D64JHW36T5LGBYG6IDZVYQD4GZWCYQI55LR4QOHOPCDRTUJJHMYA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.568503540 (5h48m5.155559782s) [noise=true , address=104.36.82.6:4873 , piece_id=D5QPVOWRUSHU5HITFOBHNYS3J7REL5XOROWGLJC4EAHXNH7EJ73A, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.576217691 (5h48m5.163273931s) [noise=true , address=45.91.116.12:4226 , piece_id=2YPECU45KNEFQEABDWQNUY6DTGZZWUFE5YK767DQSZKXW6AHJBDA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.578272647 (5h48m5.165328888s) [noise=true , address=77.90.31.17:4770 , piece_id=EGU57FLC2PJOV3AXAK2NDGYGHISFH3L3GFK5ZMWUM45MG3RUUOOA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.735239817 (5h48m5.322296068s) [noise=true , address=104.36.82.16:4294 , piece_id=GM6LYNFPDJVGPTJA7ZIQYZFEIMNSCQYRSNO4TPWYZJJ6S6N63HNA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.735279151 (5h48m5.322335392s) [seg_pos={0 0}, upload=5955]: Segment reached optimal threshold of 13 pieces. Aug 7 22:14:51.735633281 (5h48m5.322689532s) [noise=true , address=77.90.31.17:4772 , piece_id=WLFZC5FRPMYYQDN4JZVPYDCNHDMVMNL2AKV4F5UKQQ7JM4JYXDFA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection; write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection Aug 7 22:14:51.735644341 (5h48m5.322700583s) [noise=true , address=77.90.31.16:4735 , piece_id=HQ2BPD52ZQXRH5YHMX7JGS2JJ5I5BQALG327UMWAYUW7W245N4QQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection Aug 7 22:14:51.735698282 (5h48m5.322754524s) [noise=true , address=77.90.31.13:4626 , piece_id=4WWEMP7VRA72YTDSGUSQHVACDKAPJNH5JO3XJQQDS5A3VY7HLENA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection; write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection Aug 7 22:14:51.735750810 (5h48m5.322807052s) [seg_pos={0 0}, upload=5955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:14:51.735825741 (5h48m5.322881982s) [upload=5955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:14:51.735831010 (5h48m5.322887252s) [upload=5955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:28.158244845 (5h48m41.745301096s) [upload=5955]: Done uploading segments... Aug 7 22:15:28.158274540 (5h48m41.74533079s) [upload=5955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:28.250312080 (5h48m41.837368322s) [upload=5969]: Starting upload Aug 7 22:15:28.250412248 (5h48m41.837468498s) [upload=5969]: Done starting upload Aug 7 22:15:28.250539435 (5h48m41.837595676s) [upload=5969]: Uploading segments... Aug 7 22:15:28.250814920 (5h48m41.83787116s) [upload=5969]: Got next segment. Inline: false Aug 7 22:15:28.250863259 (5h48m41.8379195s) [upload=5969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:28.250871685 (5h48m41.837927936s) [upload=5969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:58.237095607 (5h49m11.824151818s) [upload=5969]: Next segment err: context deadline exceeded Aug 7 22:15:58.237308174 (5h49m11.824364395s) [upload=5969]: Done uploading segments... Aug 7 22:15:58.237357947 (5h49m11.824414157s) [upload=5969]: Upload finished. err: context deadline exceeded Aug 7 22:16:42.428984950 (5h49m56.016041171s) [upload=5985]: Starting upload Aug 7 22:16:42.429041335 (5h49m56.016097556s) [upload=5985]: Done starting upload Aug 7 22:16:42.429149126 (5h49m56.016205347s) [upload=5985]: Uploading segments... Aug 7 22:16:42.429280742 (5h49m56.016336963s) [upload=5985]: Got next segment. Inline: false Aug 7 22:16:42.429312962 (5h49m56.016369183s) [upload=5985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:42.429316729 (5h49m56.01637295s) [upload=5985]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:12.410858597 (5h50m25.997914808s) [upload=5985]: Next segment err: context deadline exceeded Aug 7 22:17:12.411035658 (5h50m25.998091879s) [upload=5985]: Done uploading segments... Aug 7 22:17:12.411065273 (5h50m25.998121494s) [upload=5985]: Upload finished. err: context deadline exceeded Duration: 30.00052854s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 122.425221ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:17:08 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 122.443265ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.X.XX...............................XXXX] duration: 1m0.000822315s FAIL (2025-08-07 22:17:04 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 22:06:14.892558219 (5h39m28.47961446s) [upload=5816]: Done uploading segments... Aug 7 22:06:14.892577646 (5h39m28.479633886s) [upload=5816]: Upload finished. err: <nil> Aug 7 22:06:56.585938978 (5h40m10.172995189s) [upload=5826]: Starting upload Aug 7 22:06:56.585989272 (5h40m10.173045493s) [upload=5826]: Done starting upload Aug 7 22:06:56.586034606 (5h40m10.173090827s) [upload=5826]: Uploading segments... Aug 7 22:06:56.586059122 (5h40m10.173115343s) [upload=5826]: Got next segment. Inline: true Aug 7 22:06:56.586062599 (5h40m10.173118809s) [upload=5826]: Waiting for error group managing segments... Aug 7 22:06:56.586092645 (5h40m10.173148866s) [upload=5826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:56.586096352 (5h40m10.173152573s) [upload=5826]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:56.586098696 (5h40m10.173154917s) [upload=5826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:56.650613061 (5h40m10.237669272s) [upload=5826]: Done uploading segments... Aug 7 22:06:56.650627668 (5h40m10.237683889s) [upload=5826]: Upload finished. err: <nil> Aug 7 22:07:58.192526348 (5h41m11.77958259s) [upload=5845]: Starting upload Aug 7 22:07:58.192614623 (5h41m11.779670864s) [upload=5845]: Done starting upload Aug 7 22:07:58.192700764 (5h41m11.779757014s) [upload=5845]: Uploading segments... Aug 7 22:07:58.192744676 (5h41m11.779800917s) [upload=5845]: Got next segment. Inline: true Aug 7 22:07:58.192750096 (5h41m11.779806337s) [upload=5845]: Waiting for error group managing segments... Aug 7 22:07:58.192795772 (5h41m11.779852013s) [upload=5845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:58.192801672 (5h41m11.779857913s) [upload=5845]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:58.192805169 (5h41m11.779861409s) [upload=5845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:58.264015693 (5h41m11.851071934s) [upload=5845]: Done uploading segments... Aug 7 22:07:58.264038786 (5h41m11.851095027s) [upload=5845]: Upload finished. err: <nil> Aug 7 22:09:12.971535319 (5h42m26.558591531s) [upload=5870]: Starting upload Aug 7 22:09:12.971593908 (5h42m26.558650129s) [upload=5870]: Done starting upload Aug 7 22:09:12.971640085 (5h42m26.558696306s) [upload=5870]: Uploading segments... Aug 7 22:09:12.971667336 (5h42m26.558723547s) [upload=5870]: Got next segment. Inline: true Aug 7 22:09:12.971670973 (5h42m26.558727184s) [upload=5870]: Waiting for error group managing segments... Aug 7 22:09:12.971701830 (5h42m26.558758051s) [upload=5870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:12.971705557 (5h42m26.558761768s) [upload=5870]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:12.971707941 (5h42m26.558764152s) [upload=5870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:13.026541326 (5h42m26.613597568s) [upload=5870]: Done uploading segments... Aug 7 22:09:13.026558519 (5h42m26.61361477s) [upload=5870]: Upload finished. err: <nil> Aug 7 22:10:14.601261519 (5h43m28.188317731s) [upload=5887]: Starting upload Aug 7 22:10:14.601309920 (5h43m28.188366141s) [upload=5887]: Done starting upload Aug 7 22:10:14.601350215 (5h43m28.188406436s) [upload=5887]: Uploading segments... Aug 7 22:10:14.601378688 (5h43m28.188434909s) [upload=5887]: Got next segment. Inline: true Aug 7 22:10:14.601382325 (5h43m28.188438546s) [upload=5887]: Waiting for error group managing segments... Aug 7 22:10:14.601411149 (5h43m28.188467361s) [upload=5887]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:14.601414947 (5h43m28.188471168s) [upload=5887]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:14.601417170 (5h43m28.188473391s) [upload=5887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:14.743486813 (5h43m28.330543034s) [upload=5887]: Done uploading segments... Aug 7 22:10:14.743500428 (5h43m28.33055664s) [upload=5887]: Upload finished. err: <nil> Aug 7 22:11:00.429392635 (5h44m14.016448856s) [upload=5899]: Starting upload Aug 7 22:11:00.429436647 (5h44m14.016492868s) [upload=5899]: Done starting upload Aug 7 22:11:00.429474658 (5h44m14.016530879s) [upload=5899]: Uploading segments... Aug 7 22:11:00.429501167 (5h44m14.016557378s) [upload=5899]: Got next segment. Inline: true Aug 7 22:11:00.429504493 (5h44m14.016560714s) [upload=5899]: Waiting for error group managing segments... Aug 7 22:11:00.429533688 (5h44m14.016589909s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:00.429537505 (5h44m14.016593717s) [upload=5899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:00.429540060 (5h44m14.016596281s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:00.493866886 (5h44m14.080923097s) [upload=5899]: Done uploading segments... Aug 7 22:11:00.493879859 (5h44m14.08093608s) [upload=5899]: Upload finished. err: <nil> Aug 7 22:12:13.235953272 (5h45m26.823009493s) [upload=5915]: Starting upload Aug 7 22:12:13.236008135 (5h45m26.823064356s) [upload=5915]: Done starting upload Aug 7 22:12:13.236053961 (5h45m26.823110172s) [upload=5915]: Uploading segments... Aug 7 22:12:13.236083775 (5h45m26.823139997s) [upload=5915]: Got next segment. Inline: true Aug 7 22:12:13.236087523 (5h45m26.823143733s) [upload=5915]: Waiting for error group managing segments... Aug 7 22:12:13.236120344 (5h45m26.823176555s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:13.236123931 (5h45m26.823180142s) [upload=5915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:13.236126285 (5h45m26.823182506s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:13.466422283 (5h45m27.053478504s) [upload=5915]: Done uploading segments... Aug 7 22:12:13.466437992 (5h45m27.053494213s) [upload=5915]: Upload finished. err: <nil> Aug 7 22:12:59.930261536 (5h46m13.517317778s) [upload=5926]: Starting upload Aug 7 22:12:59.930344330 (5h46m13.517400582s) [upload=5926]: Done starting upload Aug 7 22:12:59.930411596 (5h46m13.517467837s) [upload=5926]: Uploading segments... Aug 7 22:12:59.930461800 (5h46m13.517518041s) [upload=5926]: Got next segment. Inline: true Aug 7 22:12:59.930467751 (5h46m13.517523992s) [upload=5926]: Waiting for error group managing segments... Aug 7 22:12:59.930514368 (5h46m13.51757061s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:59.930520589 (5h46m13.51757683s) [upload=5926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:59.930524356 (5h46m13.517580607s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:59.913184211 (5h47m13.500240423s) [upload=5926]: Done uploading segments... Aug 7 22:13:59.913207345 (5h47m13.500263566s) [upload=5926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:00.010650606 (5h47m13.597706828s) [upload=5944]: Starting upload Aug 7 22:14:00.010701831 (5h47m13.597758043s) [upload=5944]: Done starting upload Aug 7 22:14:00.010753608 (5h47m13.59780983s) [upload=5944]: Uploading segments... Aug 7 22:14:00.010809803 (5h47m13.597866024s) [upload=5944]: Got next segment. Inline: true Aug 7 22:14:00.010813610 (5h47m13.597869821s) [upload=5944]: Waiting for error group managing segments... Aug 7 22:14:00.010845560 (5h47m13.597901781s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:00.010849677 (5h47m13.597905899s) [upload=5944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:00.010851892 (5h47m13.597908113s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:59.990902753 (5h48m13.577958974s) [upload=5944]: Done uploading segments... Aug 7 22:14:59.990918592 (5h48m13.577974803s) [upload=5944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:04.091713143 (5h48m17.678769415s) [upload=5961]: Starting upload Aug 7 22:15:04.091810745 (5h48m17.678866986s) [upload=5961]: Done starting upload Aug 7 22:15:04.091875977 (5h48m17.678932217s) [upload=5961]: Uploading segments... Aug 7 22:15:04.091965724 (5h48m17.679021975s) [upload=5961]: Got next segment. Inline: true Aug 7 22:15:04.091976223 (5h48m17.679032475s) [upload=5961]: Waiting for error group managing segments... Aug 7 22:15:04.092046285 (5h48m17.679102526s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:04.092055141 (5h48m17.679111383s) [upload=5961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:04.092061392 (5h48m17.679117633s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.079406942 (5h49m17.666463153s) [upload=5961]: Done uploading segments... Aug 7 22:16:04.079423422 (5h49m17.666479644s) [upload=5961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:04.164640982 (5h49m17.751697203s) [upload=5974]: Starting upload Aug 7 22:16:04.164694422 (5h49m17.751750633s) [upload=5974]: Done starting upload Aug 7 22:16:04.164734035 (5h49m17.751790246s) [upload=5974]: Uploading segments... Aug 7 22:16:04.164772067 (5h49m17.751828288s) [upload=5974]: Got next segment. Inline: true Aug 7 22:16:04.164776555 (5h49m17.751832776s) [upload=5974]: Waiting for error group managing segments... Aug 7 22:16:04.164825836 (5h49m17.751882047s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:04.164830154 (5h49m17.751886375s) [upload=5974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:04.164833621 (5h49m17.751889842s) [upload=5974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:04.148146721 (5h50m17.735202963s) [upload=5974]: Done uploading segments... Aug 7 22:17:04.148198359 (5h50m17.735254599s) [upload=5974]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000841751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 169.879183ms FAIL (2025-08-07 22:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 169.901355ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXX.............................XXXXXXXXX] duration: 30.001944271s FAIL (2025-08-07 22:16:57 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=5877]: Upload finished. err: <nil> Aug 7 22:10:15.358455928 (5h43m28.945512149s) [upload=5888]: Starting upload Aug 7 22:10:15.358522273 (5h43m28.945578494s) [upload=5888]: Done starting upload Aug 7 22:10:15.358588375 (5h43m28.945644586s) [upload=5888]: Uploading segments... Aug 7 22:10:15.358619654 (5h43m28.945675866s) [upload=5888]: Got next segment. Inline: true Aug 7 22:10:15.358623641 (5h43m28.945679853s) [upload=5888]: Waiting for error group managing segments... Aug 7 22:10:15.358657604 (5h43m28.945713826s) [upload=5888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:15.358661782 (5h43m28.945717993s) [upload=5888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:15.358665169 (5h43m28.94572139s) [upload=5888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:15.541784918 (5h43m29.128841169s) [upload=5888]: Done uploading segments... Aug 7 22:10:15.541804064 (5h43m29.128860304s) [upload=5888]: Upload finished. err: <nil> Aug 7 22:11:10.710381989 (5h44m24.29743823s) [upload=5902]: Starting upload Aug 7 22:11:10.710469673 (5h44m24.297525913s) [upload=5902]: Done starting upload Aug 7 22:11:10.710537730 (5h44m24.297593971s) [upload=5902]: Uploading segments... Aug 7 22:11:10.710600456 (5h44m24.297656698s) [upload=5902]: Got next segment. Inline: true Aug 7 22:11:10.710607830 (5h44m24.297664072s) [upload=5902]: Waiting for error group managing segments... Aug 7 22:11:10.710658876 (5h44m24.297715118s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:10.710666100 (5h44m24.297722341s) [upload=5902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:10.710670057 (5h44m24.297726299s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:40.695011486 (5h44m54.282067737s) [upload=5902]: Done uploading segments... Aug 7 22:11:40.695031373 (5h44m54.282087624s) [upload=5902]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:11:40.801935294 (5h44m54.388991515s) [upload=5908]: Starting upload Aug 7 22:11:40.802009452 (5h44m54.389065673s) [upload=5908]: Done starting upload Aug 7 22:11:40.802061539 (5h44m54.38911776s) [upload=5908]: Uploading segments... Aug 7 22:11:40.802108137 (5h44m54.389164358s) [upload=5908]: Got next segment. Inline: true Aug 7 22:11:40.802112685 (5h44m54.389168896s) [upload=5908]: Waiting for error group managing segments... Aug 7 22:11:40.802146128 (5h44m54.389202338s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.802150235 (5h44m54.389206456s) [upload=5908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:40.802172947 (5h44m54.389229168s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.785746890 (5h45m24.372803111s) [upload=5908]: Done uploading segments... Aug 7 22:12:10.785765736 (5h45m24.372821946s) [upload=5908]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:34.227606352 (5h45m47.814662563s) [upload=5920]: Starting upload Aug 7 22:12:34.227679809 (5h45m47.81473602s) [upload=5920]: Done starting upload Aug 7 22:12:34.227726427 (5h45m47.814782637s) [upload=5920]: Uploading segments... Aug 7 22:12:34.227755791 (5h45m47.814812012s) [upload=5920]: Got next segment. Inline: true Aug 7 22:12:34.227759368 (5h45m47.814815579s) [upload=5920]: Waiting for error group managing segments... Aug 7 22:12:34.227789123 (5h45m47.814845344s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:34.227792860 (5h45m47.814849081s) [upload=5920]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:34.227795325 (5h45m47.814851546s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:04.214429951 (5h46m17.801486172s) [upload=5920]: Done uploading segments... Aug 7 22:13:04.214444368 (5h46m17.801500579s) [upload=5920]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:21.693003516 (5h46m35.280059727s) [upload=5934]: Starting upload Aug 7 22:13:21.693054191 (5h46m35.280110402s) [upload=5934]: Done starting upload Aug 7 22:13:21.693097442 (5h46m35.280153663s) [upload=5934]: Uploading segments... Aug 7 22:13:21.693124161 (5h46m35.280180382s) [upload=5934]: Got next segment. Inline: true Aug 7 22:13:21.693127627 (5h46m35.280183848s) [upload=5934]: Waiting for error group managing segments... Aug 7 22:13:21.693172341 (5h46m35.280228552s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:21.693176398 (5h46m35.28023262s) [upload=5934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:21.693178873 (5h46m35.280235085s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:51.677769243 (5h47m5.264825484s) [upload=5934]: Done uploading segments... Aug 7 22:13:51.677786175 (5h47m5.264842416s) [upload=5934]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:51.763566605 (5h47m5.350622826s) [upload=5942]: Starting upload Aug 7 22:13:51.763615116 (5h47m5.350671337s) [upload=5942]: Done starting upload Aug 7 22:13:51.763654910 (5h47m5.350711131s) [upload=5942]: Uploading segments... Aug 7 22:13:51.763683413 (5h47m5.350739634s) [upload=5942]: Got next segment. Inline: true Aug 7 22:13:51.763687000 (5h47m5.350743211s) [upload=5942]: Waiting for error group managing segments... Aug 7 22:13:51.763718620 (5h47m5.350774841s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.763722296 (5h47m5.350778517s) [upload=5942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:51.763724580 (5h47m5.350780801s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:21.752690510 (5h47m35.339746731s) [upload=5942]: Done uploading segments... Aug 7 22:14:21.752712410 (5h47m35.339768631s) [upload=5942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:21.832761353 (5h47m35.419817564s) [upload=5952]: Starting upload Aug 7 22:14:21.832808701 (5h47m35.419864913s) [upload=5952]: Done starting upload Aug 7 22:14:21.832873814 (5h47m35.419930035s) [upload=5952]: Uploading segments... Aug 7 22:14:21.832905782 (5h47m35.419961993s) [upload=5952]: Got next segment. Inline: true Aug 7 22:14:21.832909440 (5h47m35.419965661s) [upload=5952]: Waiting for error group managing segments... Aug 7 22:14:21.832940607 (5h47m35.419996828s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:21.832944185 (5h47m35.420000406s) [upload=5952]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:21.832946779 (5h47m35.420003s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:51.821960613 (5h48m5.409016855s) [upload=5952]: Done uploading segments... Aug 7 22:14:51.821995769 (5h48m5.40905202s) [upload=5952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:07.153250299 (5h48m20.74030654s) [upload=5963]: Starting upload Aug 7 22:15:07.153363971 (5h48m20.740420213s) [upload=5963]: Done starting upload Aug 7 22:15:07.153422571 (5h48m20.740478811s) [upload=5963]: Uploading segments... Aug 7 22:15:07.153473596 (5h48m20.740529837s) [upload=5963]: Got next segment. Inline: true Aug 7 22:15:07.153480569 (5h48m20.74053681s) [upload=5963]: Waiting for error group managing segments... Aug 7 22:15:07.153529239 (5h48m20.740585481s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:07.153535742 (5h48m20.740591973s) [upload=5963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:07.153539228 (5h48m20.740595469s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:37.139229671 (5h48m50.726285892s) [upload=5963]: Done uploading segments... Aug 7 22:15:37.139243938 (5h48m50.726300148s) [upload=5963]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:37.225059294 (5h48m50.812115505s) [upload=5970]: Starting upload Aug 7 22:15:37.225107253 (5h48m50.812163474s) [upload=5970]: Done starting upload Aug 7 22:15:37.225145674 (5h48m50.812201895s) [upload=5970]: Uploading segments... Aug 7 22:15:37.225193384 (5h48m50.812249605s) [upload=5970]: Got next segment. Inline: true Aug 7 22:15:37.225196900 (5h48m50.81225311s) [upload=5970]: Waiting for error group managing segments... Aug 7 22:15:37.225226886 (5h48m50.812283107s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:37.225231044 (5h48m50.812287254s) [upload=5970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:37.225233308 (5h48m50.812289529s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:07.214043839 (5h49m20.80110009s) [upload=5970]: Done uploading segments... Aug 7 22:16:07.214071880 (5h49m20.801128131s) [upload=5970]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:27.239225384 (5h49m40.826281605s) [upload=5983]: Starting upload Aug 7 22:16:27.239274204 (5h49m40.826330425s) [upload=5983]: Done starting upload Aug 7 22:16:27.239318858 (5h49m40.826375079s) [upload=5983]: Uploading segments... Aug 7 22:16:27.239353452 (5h49m40.826409673s) [upload=5983]: Got next segment. Inline: true Aug 7 22:16:27.239358842 (5h49m40.826415063s) [upload=5983]: Waiting for error group managing segments... Aug 7 22:16:27.239399728 (5h49m40.82645595s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:27.239405820 (5h49m40.826462041s) [upload=5983]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:27.239410048 (5h49m40.826466269s) [upload=5983]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:57.228025199 (5h50m10.81508144s) [upload=5983]: Done uploading segments... Aug 7 22:16:57.228050357 (5h50m10.815106607s) [upload=5983]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001966322s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...X.....X..............................XXXXXXX] duration: 30.001238638s FAIL (2025-08-07 22:16:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001249847s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXX.XX.........................................XXXXXX] duration: 30.000580308s FAIL (2025-08-07 22:16:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000600356s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.X..................................XXXXXXXX] duration: 30.001771188s FAIL (2025-08-07 22:16:48 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001789962s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX....................................XXXXX] duration: 1m0.000486562s FAIL (2025-08-07 22:16:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000499737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XX......X.............................XXXXX] duration: 1m0.000575907s FAIL (2025-08-07 22:16:22 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== EZDFZXS3M53PGM7LOPDNB3ARCXNGCNKOJQGDEY4Q, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.367401928 (5h43m13.954458149s) [noise=true , address=104.36.82.59:4323 , piece_id=RDLGI7JXFHQINYTMPXTQLIREY6SOQJJ4YIJSCUG3UEZWXLWNGMNQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.368760244 (5h43m13.955816485s) [noise=true , address=104.36.82.59:4309 , piece_id=YYKRP337BNKDZYEEK5P7MDA62ELU5VGRSRB6ML76DKG6XYQS5QGQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.390444030 (5h43m13.977500241s) [noise=true , address=104.36.82.55:4212 , piece_id=R44VQAJ47VZZ622QQSXSS6ZLYJVQOYHVJFBK5XLLPHDLTPZG3OQA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.391147313 (5h43m13.978203534s) [noise=true , address=104.36.82.15:4268 , piece_id=EFSX37BH3XS3HWGET5VD3RYRGJWJK4PDGXG45IVJKB7SOSVP7YZA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.399754149 (5h43m13.98681037s) [noise=true , address=104.36.82.9:4011 , piece_id=UT6I3AJEWBCSPWUJPGJK3BUB6SZMFAZ5EQBAQ56Q65BFAAIIROIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.460854191 (5h43m14.047910432s) [noise=true , address=104.36.82.42:4799 , piece_id=LXQBRMRAEKV5OY2L5WGUW4E43RMLFBHYZRTZ2IUGMUDENRDBQLFQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.470345858 (5h43m14.0574021s) [noise=true , address=104.36.82.43:4846 , piece_id=3HN5X5L6FZOM2LY6UG6FUPFQSJGVCISOHFDI4T6IIW4XRCV7EKCQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.507622690 (5h43m14.094678901s) [noise=true , address=104.36.82.45:4913 , piece_id=FN5X5ZS7L7HRERGTFSBHDONB5V57E2AE5RG55CEAS6WPI6L7VWFQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.534357043 (5h43m14.121413254s) [noise=true , address=104.36.82.58:4279 , piece_id=Z2FXWDDE6Q4S5P73GMZFI6RM5LBBRWMDLKH6XK7MF3IAUWZC6PIA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.552361511 (5h43m14.139417732s) [noise=true , address=104.36.82.4:4827 , piece_id=QP6JUIBNRV7YFX433AN5PLRGTRPQQ2XC52HF7EYQSGF3FMAXM4JA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.596411237 (5h43m14.183467448s) [noise=true , address=104.36.82.56:4241 , piece_id=CZNEIWPUR5RWWSRD42ZZV76SKKYQUL22K5OSMRABZ3JLNWPOQQPQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.596432907 (5h43m14.183489128s) [seg_pos={0 0}, upload=5882]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:00.596673115 (5h43m14.183729336s) [noise=true , address=104.36.82.13:4197 , piece_id=YXUCKQYQYWVKLTOR6AZHGS5LANJQESFHAGE6F2ZA272CZDT2BUIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection; write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection Aug 7 22:10:00.596697370 (5h43m14.183753591s) [noise=true , address=104.36.82.16:4327 , piece_id=GRKTSALMEB73JLA74SJ6ET75F3JY4PXVTYOGTANJBS3VOETOVBIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection; write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection Aug 7 22:10:00.596712248 (5h43m14.183768469s) [noise=true , address=79.127.205.234:4600 , piece_id=LP74NCGX4H2E4U5UHJATTAARN6E4O4MVSVP2ED2N67LXJQX7DPAQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kxaNeRZqxza8AxSaF4PvBkmKoUY2NvficDEBRaXfDnq1HSgJ1): protocol: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection; write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection Aug 7 22:10:00.596771750 (5h43m14.183827971s) [noise=true , address=104.36.82.37:4685 , piece_id=4Q4WHVJWI4VXAEZEPTS355RHEBI37ME5S3MDUI5KM45LVDZ3GZ3A, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MhzeNvbKYhZVnhQW8oY5BeGQDuyNtmi25Xhgzrej4bCyoD9VM): protocol: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection Aug 7 22:10:00.596795714 (5h43m14.183851935s) [seg_pos={0 0}, upload=5882]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:00.596866456 (5h43m14.183922677s) [upload=5882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:00.596873089 (5h43m14.18392931s) [upload=5882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:00.660018838 (5h43m14.247075089s) [upload=5882]: Done uploading segments... Aug 7 22:10:00.660039907 (5h43m14.247096149s) [upload=5882]: Upload finished. err: <nil> Aug 7 22:11:14.838405386 (5h44m28.425461627s) [upload=5904]: Starting upload Aug 7 22:11:14.838502537 (5h44m28.425558788s) [upload=5904]: Done starting upload Aug 7 22:11:14.838673375 (5h44m28.425729627s) [upload=5904]: Uploading segments... Aug 7 22:11:14.838929564 (5h44m28.425985815s) [upload=5904]: Got next segment. Inline: false Aug 7 22:11:14.838997871 (5h44m28.426054112s) [upload=5904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:14.839003762 (5h44m28.426060003s) [upload=5904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:14.826003908 (5h45m28.413060119s) [upload=5904]: Next segment err: context deadline exceeded Aug 7 22:12:14.826061727 (5h45m28.413117948s) [upload=5904]: Done uploading segments... Aug 7 22:12:14.826077997 (5h45m28.413134218s) [upload=5904]: Upload finished. err: context deadline exceeded Aug 7 22:12:14.896180564 (5h45m28.483236775s) [upload=5918]: Starting upload Aug 7 22:12:14.896231629 (5h45m28.483287841s) [upload=5918]: Done starting upload Aug 7 22:12:14.896288415 (5h45m28.483344636s) [upload=5918]: Uploading segments... Aug 7 22:12:14.896472659 (5h45m28.48352888s) [upload=5918]: Got next segment. Inline: false Aug 7 22:12:14.896503706 (5h45m28.483559918s) [upload=5918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:14.896507904 (5h45m28.483564126s) [upload=5918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:14.884243492 (5h46m28.471299734s) [upload=5918]: Next segment err: context deadline exceeded Aug 7 22:13:14.884316208 (5h46m28.471372449s) [upload=5918]: Done uploading segments... Aug 7 22:13:14.884334913 (5h46m28.471391154s) [upload=5918]: Upload finished. err: context deadline exceeded Aug 7 22:13:15.003888637 (5h46m28.590944898s) [upload=5932]: Starting upload Aug 7 22:13:15.004039338 (5h46m28.591095609s) [upload=5932]: Done starting upload Aug 7 22:13:15.004748231 (5h46m28.591804483s) [upload=5932]: Uploading segments... Aug 7 22:13:15.004838500 (5h46m28.591894741s) [upload=5932]: Got next segment. Inline: false Aug 7 22:13:15.004895547 (5h46m28.591951788s) [upload=5932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:15.004902439 (5h46m28.591958681s) [upload=5932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:14.988056084 (5h47m28.575112325s) [upload=5932]: Next segment err: context deadline exceeded Aug 7 22:14:14.988128109 (5h47m28.575184351s) [upload=5932]: Done uploading segments... Aug 7 22:14:14.988155329 (5h47m28.57521157s) [upload=5932]: Upload finished. err: context deadline exceeded Aug 7 22:14:22.127816864 (5h47m35.714873075s) [upload=5953]: Starting upload Aug 7 22:14:22.127868040 (5h47m35.714924261s) [upload=5953]: Done starting upload Aug 7 22:14:22.127944151 (5h47m35.715000372s) [upload=5953]: Uploading segments... Aug 7 22:14:22.128112817 (5h47m35.715169038s) [upload=5953]: Got next segment. Inline: false Aug 7 22:14:22.128145027 (5h47m35.715201248s) [upload=5953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:22.128148994 (5h47m35.715205205s) [upload=5953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:22.111818521 (5h48m35.698874742s) [upload=5953]: Next segment err: context deadline exceeded Aug 7 22:15:22.111891768 (5h48m35.698947999s) [upload=5953]: Done uploading segments... Aug 7 22:15:22.111945839 (5h48m35.69900206s) [upload=5953]: Upload finished. err: context deadline exceeded Aug 7 22:15:22.210077596 (5h48m35.797133807s) [upload=5967]: Starting upload Aug 7 22:15:22.210126076 (5h48m35.797182286s) [upload=5967]: Done starting upload Aug 7 22:15:22.210213909 (5h48m35.79727013s) [upload=5967]: Uploading segments... Aug 7 22:15:22.210350815 (5h48m35.797407026s) [upload=5967]: Got next segment. Inline: false Aug 7 22:15:22.210380320 (5h48m35.797436542s) [upload=5967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:22.210384398 (5h48m35.797440619s) [upload=5967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:22.196775688 (5h49m35.783831939s) [upload=5967]: Next segment err: context deadline exceeded Aug 7 22:16:22.196861187 (5h49m35.783917439s) [upload=5967]: Done uploading segments... Aug 7 22:16:22.196885322 (5h49m35.783941574s) [upload=5967]: Upload finished. err: context deadline exceeded Duration: 1m0.000622565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXX.............................XXXXXXXX] duration: 30.000814759s FAIL (2025-08-07 22:16:07 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6889 (5h42m18.223413121s) [upload=5866]: Upload finished. err: <nil> Aug 7 22:09:43.052830170 (5h42m56.639886381s) [upload=5877]: Starting upload Aug 7 22:09:43.052883700 (5h42m56.639939921s) [upload=5877]: Done starting upload Aug 7 22:09:43.052934405 (5h42m56.639990626s) [upload=5877]: Uploading segments... Aug 7 22:09:43.052967446 (5h42m56.640023657s) [upload=5877]: Got next segment. Inline: true Aug 7 22:09:43.052971313 (5h42m56.640027534s) [upload=5877]: Waiting for error group managing segments... Aug 7 22:09:43.053000388 (5h42m56.640056609s) [upload=5877]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:43.053004064 (5h42m56.640060276s) [upload=5877]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:43.053006990 (5h42m56.640063211s) [upload=5877]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:43.127325938 (5h42m56.714382178s) [upload=5877]: Done uploading segments... Aug 7 22:09:43.127344483 (5h42m56.714400723s) [upload=5877]: Upload finished. err: <nil> Aug 7 22:10:15.358455928 (5h43m28.945512149s) [upload=5888]: Starting upload Aug 7 22:10:15.358522273 (5h43m28.945578494s) [upload=5888]: Done starting upload Aug 7 22:10:15.358588375 (5h43m28.945644586s) [upload=5888]: Uploading segments... Aug 7 22:10:15.358619654 (5h43m28.945675866s) [upload=5888]: Got next segment. Inline: true Aug 7 22:10:15.358623641 (5h43m28.945679853s) [upload=5888]: Waiting for error group managing segments... Aug 7 22:10:15.358657604 (5h43m28.945713826s) [upload=5888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:15.358661782 (5h43m28.945717993s) [upload=5888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:15.358665169 (5h43m28.94572139s) [upload=5888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:15.541784918 (5h43m29.128841169s) [upload=5888]: Done uploading segments... Aug 7 22:10:15.541804064 (5h43m29.128860304s) [upload=5888]: Upload finished. err: <nil> Aug 7 22:11:10.710381989 (5h44m24.29743823s) [upload=5902]: Starting upload Aug 7 22:11:10.710469673 (5h44m24.297525913s) [upload=5902]: Done starting upload Aug 7 22:11:10.710537730 (5h44m24.297593971s) [upload=5902]: Uploading segments... Aug 7 22:11:10.710600456 (5h44m24.297656698s) [upload=5902]: Got next segment. Inline: true Aug 7 22:11:10.710607830 (5h44m24.297664072s) [upload=5902]: Waiting for error group managing segments... Aug 7 22:11:10.710658876 (5h44m24.297715118s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:10.710666100 (5h44m24.297722341s) [upload=5902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:10.710670057 (5h44m24.297726299s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:40.695011486 (5h44m54.282067737s) [upload=5902]: Done uploading segments... Aug 7 22:11:40.695031373 (5h44m54.282087624s) [upload=5902]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:11:40.801935294 (5h44m54.388991515s) [upload=5908]: Starting upload Aug 7 22:11:40.802009452 (5h44m54.389065673s) [upload=5908]: Done starting upload Aug 7 22:11:40.802061539 (5h44m54.38911776s) [upload=5908]: Uploading segments... Aug 7 22:11:40.802108137 (5h44m54.389164358s) [upload=5908]: Got next segment. Inline: true Aug 7 22:11:40.802112685 (5h44m54.389168896s) [upload=5908]: Waiting for error group managing segments... Aug 7 22:11:40.802146128 (5h44m54.389202338s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.802150235 (5h44m54.389206456s) [upload=5908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:40.802172947 (5h44m54.389229168s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.785746890 (5h45m24.372803111s) [upload=5908]: Done uploading segments... Aug 7 22:12:10.785765736 (5h45m24.372821946s) [upload=5908]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:34.227606352 (5h45m47.814662563s) [upload=5920]: Starting upload Aug 7 22:12:34.227679809 (5h45m47.81473602s) [upload=5920]: Done starting upload Aug 7 22:12:34.227726427 (5h45m47.814782637s) [upload=5920]: Uploading segments... Aug 7 22:12:34.227755791 (5h45m47.814812012s) [upload=5920]: Got next segment. Inline: true Aug 7 22:12:34.227759368 (5h45m47.814815579s) [upload=5920]: Waiting for error group managing segments... Aug 7 22:12:34.227789123 (5h45m47.814845344s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:34.227792860 (5h45m47.814849081s) [upload=5920]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:34.227795325 (5h45m47.814851546s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:04.214429951 (5h46m17.801486172s) [upload=5920]: Done uploading segments... Aug 7 22:13:04.214444368 (5h46m17.801500579s) [upload=5920]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:21.693003516 (5h46m35.280059727s) [upload=5934]: Starting upload Aug 7 22:13:21.693054191 (5h46m35.280110402s) [upload=5934]: Done starting upload Aug 7 22:13:21.693097442 (5h46m35.280153663s) [upload=5934]: Uploading segments... Aug 7 22:13:21.693124161 (5h46m35.280180382s) [upload=5934]: Got next segment. Inline: true Aug 7 22:13:21.693127627 (5h46m35.280183848s) [upload=5934]: Waiting for error group managing segments... Aug 7 22:13:21.693172341 (5h46m35.280228552s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:21.693176398 (5h46m35.28023262s) [upload=5934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:21.693178873 (5h46m35.280235085s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:51.677769243 (5h47m5.264825484s) [upload=5934]: Done uploading segments... Aug 7 22:13:51.677786175 (5h47m5.264842416s) [upload=5934]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:51.763566605 (5h47m5.350622826s) [upload=5942]: Starting upload Aug 7 22:13:51.763615116 (5h47m5.350671337s) [upload=5942]: Done starting upload Aug 7 22:13:51.763654910 (5h47m5.350711131s) [upload=5942]: Uploading segments... Aug 7 22:13:51.763683413 (5h47m5.350739634s) [upload=5942]: Got next segment. Inline: true Aug 7 22:13:51.763687000 (5h47m5.350743211s) [upload=5942]: Waiting for error group managing segments... Aug 7 22:13:51.763718620 (5h47m5.350774841s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.763722296 (5h47m5.350778517s) [upload=5942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:51.763724580 (5h47m5.350780801s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:21.752690510 (5h47m35.339746731s) [upload=5942]: Done uploading segments... Aug 7 22:14:21.752712410 (5h47m35.339768631s) [upload=5942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:21.832761353 (5h47m35.419817564s) [upload=5952]: Starting upload Aug 7 22:14:21.832808701 (5h47m35.419864913s) [upload=5952]: Done starting upload Aug 7 22:14:21.832873814 (5h47m35.419930035s) [upload=5952]: Uploading segments... Aug 7 22:14:21.832905782 (5h47m35.419961993s) [upload=5952]: Got next segment. Inline: true Aug 7 22:14:21.832909440 (5h47m35.419965661s) [upload=5952]: Waiting for error group managing segments... Aug 7 22:14:21.832940607 (5h47m35.419996828s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:21.832944185 (5h47m35.420000406s) [upload=5952]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:21.832946779 (5h47m35.420003s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:51.821960613 (5h48m5.409016855s) [upload=5952]: Done uploading segments... Aug 7 22:14:51.821995769 (5h48m5.40905202s) [upload=5952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:07.153250299 (5h48m20.74030654s) [upload=5963]: Starting upload Aug 7 22:15:07.153363971 (5h48m20.740420213s) [upload=5963]: Done starting upload Aug 7 22:15:07.153422571 (5h48m20.740478811s) [upload=5963]: Uploading segments... Aug 7 22:15:07.153473596 (5h48m20.740529837s) [upload=5963]: Got next segment. Inline: true Aug 7 22:15:07.153480569 (5h48m20.74053681s) [upload=5963]: Waiting for error group managing segments... Aug 7 22:15:07.153529239 (5h48m20.740585481s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:07.153535742 (5h48m20.740591973s) [upload=5963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:07.153539228 (5h48m20.740595469s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:37.139229671 (5h48m50.726285892s) [upload=5963]: Done uploading segments... Aug 7 22:15:37.139243938 (5h48m50.726300148s) [upload=5963]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:37.225059294 (5h48m50.812115505s) [upload=5970]: Starting upload Aug 7 22:15:37.225107253 (5h48m50.812163474s) [upload=5970]: Done starting upload Aug 7 22:15:37.225145674 (5h48m50.812201895s) [upload=5970]: Uploading segments... Aug 7 22:15:37.225193384 (5h48m50.812249605s) [upload=5970]: Got next segment. Inline: true Aug 7 22:15:37.225196900 (5h48m50.81225311s) [upload=5970]: Waiting for error group managing segments... Aug 7 22:15:37.225226886 (5h48m50.812283107s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:37.225231044 (5h48m50.812287254s) [upload=5970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:37.225233308 (5h48m50.812289529s) [upload=5970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:07.214043839 (5h49m20.80110009s) [upload=5970]: Done uploading segments... Aug 7 22:16:07.214071880 (5h49m20.801128131s) [upload=5970]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000833022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 419.226733ms FAIL (2025-08-07 22:16:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 419.25157ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.X.XX...............................XXX] duration: 1m0.001033661s FAIL (2025-08-07 22:16:04 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : pb.BatchRequestItem_ObjectCommit Aug 7 22:05:13.937726083 (5h38m27.524782293s) [upload=5797]: Done uploading segments... Aug 7 22:05:13.937747112 (5h38m27.524803323s) [upload=5797]: Upload finished. err: <nil> Aug 7 22:06:14.837051759 (5h39m28.42410798s) [upload=5816]: Starting upload Aug 7 22:06:14.837104066 (5h39m28.424160287s) [upload=5816]: Done starting upload Aug 7 22:06:14.837167315 (5h39m28.424223526s) [upload=5816]: Uploading segments... Aug 7 22:06:14.837196950 (5h39m28.42425316s) [upload=5816]: Got next segment. Inline: true Aug 7 22:06:14.837201157 (5h39m28.424257379s) [upload=5816]: Waiting for error group managing segments... Aug 7 22:06:14.837234911 (5h39m28.424291132s) [upload=5816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:14.837239529 (5h39m28.42429575s) [upload=5816]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:14.837241954 (5h39m28.424298175s) [upload=5816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:14.892558219 (5h39m28.47961446s) [upload=5816]: Done uploading segments... Aug 7 22:06:14.892577646 (5h39m28.479633886s) [upload=5816]: Upload finished. err: <nil> Aug 7 22:06:56.585938978 (5h40m10.172995189s) [upload=5826]: Starting upload Aug 7 22:06:56.585989272 (5h40m10.173045493s) [upload=5826]: Done starting upload Aug 7 22:06:56.586034606 (5h40m10.173090827s) [upload=5826]: Uploading segments... Aug 7 22:06:56.586059122 (5h40m10.173115343s) [upload=5826]: Got next segment. Inline: true Aug 7 22:06:56.586062599 (5h40m10.173118809s) [upload=5826]: Waiting for error group managing segments... Aug 7 22:06:56.586092645 (5h40m10.173148866s) [upload=5826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:56.586096352 (5h40m10.173152573s) [upload=5826]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:56.586098696 (5h40m10.173154917s) [upload=5826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:56.650613061 (5h40m10.237669272s) [upload=5826]: Done uploading segments... Aug 7 22:06:56.650627668 (5h40m10.237683889s) [upload=5826]: Upload finished. err: <nil> Aug 7 22:07:58.192526348 (5h41m11.77958259s) [upload=5845]: Starting upload Aug 7 22:07:58.192614623 (5h41m11.779670864s) [upload=5845]: Done starting upload Aug 7 22:07:58.192700764 (5h41m11.779757014s) [upload=5845]: Uploading segments... Aug 7 22:07:58.192744676 (5h41m11.779800917s) [upload=5845]: Got next segment. Inline: true Aug 7 22:07:58.192750096 (5h41m11.779806337s) [upload=5845]: Waiting for error group managing segments... Aug 7 22:07:58.192795772 (5h41m11.779852013s) [upload=5845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:58.192801672 (5h41m11.779857913s) [upload=5845]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:58.192805169 (5h41m11.779861409s) [upload=5845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:58.264015693 (5h41m11.851071934s) [upload=5845]: Done uploading segments... Aug 7 22:07:58.264038786 (5h41m11.851095027s) [upload=5845]: Upload finished. err: <nil> Aug 7 22:09:12.971535319 (5h42m26.558591531s) [upload=5870]: Starting upload Aug 7 22:09:12.971593908 (5h42m26.558650129s) [upload=5870]: Done starting upload Aug 7 22:09:12.971640085 (5h42m26.558696306s) [upload=5870]: Uploading segments... Aug 7 22:09:12.971667336 (5h42m26.558723547s) [upload=5870]: Got next segment. Inline: true Aug 7 22:09:12.971670973 (5h42m26.558727184s) [upload=5870]: Waiting for error group managing segments... Aug 7 22:09:12.971701830 (5h42m26.558758051s) [upload=5870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:12.971705557 (5h42m26.558761768s) [upload=5870]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:12.971707941 (5h42m26.558764152s) [upload=5870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:13.026541326 (5h42m26.613597568s) [upload=5870]: Done uploading segments... Aug 7 22:09:13.026558519 (5h42m26.61361477s) [upload=5870]: Upload finished. err: <nil> Aug 7 22:10:14.601261519 (5h43m28.188317731s) [upload=5887]: Starting upload Aug 7 22:10:14.601309920 (5h43m28.188366141s) [upload=5887]: Done starting upload Aug 7 22:10:14.601350215 (5h43m28.188406436s) [upload=5887]: Uploading segments... Aug 7 22:10:14.601378688 (5h43m28.188434909s) [upload=5887]: Got next segment. Inline: true Aug 7 22:10:14.601382325 (5h43m28.188438546s) [upload=5887]: Waiting for error group managing segments... Aug 7 22:10:14.601411149 (5h43m28.188467361s) [upload=5887]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:14.601414947 (5h43m28.188471168s) [upload=5887]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:14.601417170 (5h43m28.188473391s) [upload=5887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:14.743486813 (5h43m28.330543034s) [upload=5887]: Done uploading segments... Aug 7 22:10:14.743500428 (5h43m28.33055664s) [upload=5887]: Upload finished. err: <nil> Aug 7 22:11:00.429392635 (5h44m14.016448856s) [upload=5899]: Starting upload Aug 7 22:11:00.429436647 (5h44m14.016492868s) [upload=5899]: Done starting upload Aug 7 22:11:00.429474658 (5h44m14.016530879s) [upload=5899]: Uploading segments... Aug 7 22:11:00.429501167 (5h44m14.016557378s) [upload=5899]: Got next segment. Inline: true Aug 7 22:11:00.429504493 (5h44m14.016560714s) [upload=5899]: Waiting for error group managing segments... Aug 7 22:11:00.429533688 (5h44m14.016589909s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:00.429537505 (5h44m14.016593717s) [upload=5899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:00.429540060 (5h44m14.016596281s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:00.493866886 (5h44m14.080923097s) [upload=5899]: Done uploading segments... Aug 7 22:11:00.493879859 (5h44m14.08093608s) [upload=5899]: Upload finished. err: <nil> Aug 7 22:12:13.235953272 (5h45m26.823009493s) [upload=5915]: Starting upload Aug 7 22:12:13.236008135 (5h45m26.823064356s) [upload=5915]: Done starting upload Aug 7 22:12:13.236053961 (5h45m26.823110172s) [upload=5915]: Uploading segments... Aug 7 22:12:13.236083775 (5h45m26.823139997s) [upload=5915]: Got next segment. Inline: true Aug 7 22:12:13.236087523 (5h45m26.823143733s) [upload=5915]: Waiting for error group managing segments... Aug 7 22:12:13.236120344 (5h45m26.823176555s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:13.236123931 (5h45m26.823180142s) [upload=5915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:13.236126285 (5h45m26.823182506s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:13.466422283 (5h45m27.053478504s) [upload=5915]: Done uploading segments... Aug 7 22:12:13.466437992 (5h45m27.053494213s) [upload=5915]: Upload finished. err: <nil> Aug 7 22:12:59.930261536 (5h46m13.517317778s) [upload=5926]: Starting upload Aug 7 22:12:59.930344330 (5h46m13.517400582s) [upload=5926]: Done starting upload Aug 7 22:12:59.930411596 (5h46m13.517467837s) [upload=5926]: Uploading segments... Aug 7 22:12:59.930461800 (5h46m13.517518041s) [upload=5926]: Got next segment. Inline: true Aug 7 22:12:59.930467751 (5h46m13.517523992s) [upload=5926]: Waiting for error group managing segments... Aug 7 22:12:59.930514368 (5h46m13.51757061s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:59.930520589 (5h46m13.51757683s) [upload=5926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:59.930524356 (5h46m13.517580607s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:59.913184211 (5h47m13.500240423s) [upload=5926]: Done uploading segments... Aug 7 22:13:59.913207345 (5h47m13.500263566s) [upload=5926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:00.010650606 (5h47m13.597706828s) [upload=5944]: Starting upload Aug 7 22:14:00.010701831 (5h47m13.597758043s) [upload=5944]: Done starting upload Aug 7 22:14:00.010753608 (5h47m13.59780983s) [upload=5944]: Uploading segments... Aug 7 22:14:00.010809803 (5h47m13.597866024s) [upload=5944]: Got next segment. Inline: true Aug 7 22:14:00.010813610 (5h47m13.597869821s) [upload=5944]: Waiting for error group managing segments... Aug 7 22:14:00.010845560 (5h47m13.597901781s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:00.010849677 (5h47m13.597905899s) [upload=5944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:00.010851892 (5h47m13.597908113s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:59.990902753 (5h48m13.577958974s) [upload=5944]: Done uploading segments... Aug 7 22:14:59.990918592 (5h48m13.577974803s) [upload=5944]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:04.091713143 (5h48m17.678769415s) [upload=5961]: Starting upload Aug 7 22:15:04.091810745 (5h48m17.678866986s) [upload=5961]: Done starting upload Aug 7 22:15:04.091875977 (5h48m17.678932217s) [upload=5961]: Uploading segments... Aug 7 22:15:04.091965724 (5h48m17.679021975s) [upload=5961]: Got next segment. Inline: true Aug 7 22:15:04.091976223 (5h48m17.679032475s) [upload=5961]: Waiting for error group managing segments... Aug 7 22:15:04.092046285 (5h48m17.679102526s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:04.092055141 (5h48m17.679111383s) [upload=5961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:04.092061392 (5h48m17.679117633s) [upload=5961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.079406942 (5h49m17.666463153s) [upload=5961]: Done uploading segments... Aug 7 22:16:04.079423422 (5h49m17.666479644s) [upload=5961]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.001056182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXX..X........................................XXXXXX] duration: 30.000672731s FAIL (2025-08-07 22:15:58 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== em: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:39.505507009 (5h47m53.09256325s) [upload=5955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:51.424409681 (5h48m5.011465921s) [seg_pos={0 0}, upload=5955]: Begin upload segment... Aug 7 22:14:51.424474331 (5h48m5.011530572s) [seg_pos={0 0}, upload=5955]: Done begin upload segment. Aug 7 22:14:51.424503616 (5h48m5.011559867s) [noise=true , address=104.36.82.15:4257 , piece_id=C6V3O2TFBIF5W2E4AJA6BIRK7CJOQUJ73H5SVYPB5KYXUZ3HN5PA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.424521709 (5h48m5.011577921s) [noise=true , address=45.91.116.14:4189 , piece_id=65YS2EZ7TJFMO2RHD6F2SGUN2CXHJ47AI3NOX4G45GCKR663LXTQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.424865801 (5h48m5.011922023s) [noise=true , address=45.91.116.12:4226 , piece_id=2YPECU45KNEFQEABDWQNUY6DTGZZWUFE5YK767DQSZKXW6AHJBDA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.424976539 (5h48m5.01203278s) [noise=true , address=45.91.116.12:4227 , piece_id=PT72AOCQEVQJ2ZSMYVM4FHBQSKMJDSGMKRAHRRS6W4QVTWNIYEAQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425100399 (5h48m5.01215661s) [noise=true , address=77.90.31.13:4626 , piece_id=4WWEMP7VRA72YTDSGUSQHVACDKAPJNH5JO3XJQQDS5A3VY7HLENA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425376945 (5h48m5.012433187s) [noise=true , address=104.36.82.16:4294 , piece_id=GM6LYNFPDJVGPTJA7ZIQYZFEIMNSCQYRSNO4TPWYZJJ6S6N63HNA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425848076 (5h48m5.012904316s) [noise=true , address=45.91.116.14:4190 , piece_id=D64JHW36T5LGBYG6IDZVYQD4GZWCYQI55LR4QOHOPCDRTUJJHMYA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.426300299 (5h48m5.013356541s) [noise=true , address=77.90.31.15:4697 , piece_id=XDCICPLIEUA3HBHPTL5YKQ5MITBASANALKMAXLEAATHSTA23HWBQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.426661613 (5h48m5.013717855s) [noise=true , address=45.91.116.10:4082 , piece_id=MCI5J5AEY44AJPHKEVSTVZGDBHRNMG4R5GCIASZ2OXX4F2JMIKMQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.427149574 (5h48m5.014205816s) [noise=true , address=77.90.31.17:4772 , piece_id=WLFZC5FRPMYYQDN4JZVPYDCNHDMVMNL2AKV4F5UKQQ7JM4JYXDFA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.427626695 (5h48m5.014682937s) [noise=true , address=77.90.31.16:4735 , piece_id=HQ2BPD52ZQXRH5YHMX7JGS2JJ5I5BQALG327UMWAYUW7W245N4QQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428103996 (5h48m5.015160247s) [noise=true , address=104.36.82.6:4873 , piece_id=D5QPVOWRUSHU5HITFOBHNYS3J7REL5XOROWGLJC4EAHXNH7EJ73A, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428503051 (5h48m5.015559292s) [noise=true , address=104.36.82.7:4912 , piece_id=QBFXRWNLJKWEOJ36KB6AJMOBZ64BW5NT4UZINBVGQHQZDLFXP5YA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428861140 (5h48m5.01591739s) [noise=true , address=45.91.116.5:4299 , piece_id=KP6S3WEZGKH2V5OE34LLHHSNQ5XPRKSKFNK66DVOTMWOWBJCOZQA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.429259873 (5h48m5.016316115s) [noise=true , address=45.91.116.9:4261 , piece_id=JNPPGNLOROFBT27FHNEU6HQY77NEPFQBZOHJZRDAPCUO3UOCKR2Q, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.429662896 (5h48m5.016719138s) [noise=true , address=77.90.31.17:4770 , piece_id=EGU57FLC2PJOV3AXAK2NDGYGHISFH3L3GFK5ZMWUM45MG3RUUOOA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.550946177 (5h48m5.138002409s) [upload=5955]: Next returned nil segment Aug 7 22:14:51.550961846 (5h48m5.138018078s) [upload=5955]: Waiting for error group managing segments... Aug 7 22:14:51.559757896 (5h48m5.146814117s) [noise=true , address=45.91.116.14:4189 , piece_id=65YS2EZ7TJFMO2RHD6F2SGUN2CXHJ47AI3NOX4G45GCKR663LXTQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561017878 (5h48m5.148074129s) [noise=true , address=45.91.116.12:4227 , piece_id=PT72AOCQEVQJ2ZSMYVM4FHBQSKMJDSGMKRAHRRS6W4QVTWNIYEAQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561713717 (5h48m5.148769938s) [noise=true , address=45.91.116.10:4082 , piece_id=MCI5J5AEY44AJPHKEVSTVZGDBHRNMG4R5GCIASZ2OXX4F2JMIKMQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561816609 (5h48m5.14887284s) [noise=true , address=45.91.116.5:4299 , piece_id=KP6S3WEZGKH2V5OE34LLHHSNQ5XPRKSKFNK66DVOTMWOWBJCOZQA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.562321571 (5h48m5.149377812s) [noise=true , address=45.91.116.9:4261 , piece_id=JNPPGNLOROFBT27FHNEU6HQY77NEPFQBZOHJZRDAPCUO3UOCKR2Q, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.563667614 (5h48m5.150723845s) [noise=true , address=77.90.31.15:4697 , piece_id=XDCICPLIEUA3HBHPTL5YKQ5MITBASANALKMAXLEAATHSTA23HWBQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.565443659 (5h48m5.15249991s) [noise=true , address=104.36.82.15:4257 , piece_id=C6V3O2TFBIF5W2E4AJA6BIRK7CJOQUJ73H5SVYPB5KYXUZ3HN5PA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.566941915 (5h48m5.153998147s) [noise=true , address=104.36.82.7:4912 , piece_id=QBFXRWNLJKWEOJ36KB6AJMOBZ64BW5NT4UZINBVGQHQZDLFXP5YA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.567819224 (5h48m5.154875466s) [noise=true , address=45.91.116.14:4190 , piece_id=D64JHW36T5LGBYG6IDZVYQD4GZWCYQI55LR4QOHOPCDRTUJJHMYA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.568503540 (5h48m5.155559782s) [noise=true , address=104.36.82.6:4873 , piece_id=D5QPVOWRUSHU5HITFOBHNYS3J7REL5XOROWGLJC4EAHXNH7EJ73A, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.576217691 (5h48m5.163273931s) [noise=true , address=45.91.116.12:4226 , piece_id=2YPECU45KNEFQEABDWQNUY6DTGZZWUFE5YK767DQSZKXW6AHJBDA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.578272647 (5h48m5.165328888s) [noise=true , address=77.90.31.17:4770 , piece_id=EGU57FLC2PJOV3AXAK2NDGYGHISFH3L3GFK5ZMWUM45MG3RUUOOA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.735239817 (5h48m5.322296068s) [noise=true , address=104.36.82.16:4294 , piece_id=GM6LYNFPDJVGPTJA7ZIQYZFEIMNSCQYRSNO4TPWYZJJ6S6N63HNA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.735279151 (5h48m5.322335392s) [seg_pos={0 0}, upload=5955]: Segment reached optimal threshold of 13 pieces. Aug 7 22:14:51.735633281 (5h48m5.322689532s) [noise=true , address=77.90.31.17:4772 , piece_id=WLFZC5FRPMYYQDN4JZVPYDCNHDMVMNL2AKV4F5UKQQ7JM4JYXDFA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection; write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection Aug 7 22:14:51.735644341 (5h48m5.322700583s) [noise=true , address=77.90.31.16:4735 , piece_id=HQ2BPD52ZQXRH5YHMX7JGS2JJ5I5BQALG327UMWAYUW7W245N4QQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection Aug 7 22:14:51.735698282 (5h48m5.322754524s) [noise=true , address=77.90.31.13:4626 , piece_id=4WWEMP7VRA72YTDSGUSQHVACDKAPJNH5JO3XJQQDS5A3VY7HLENA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection; write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection Aug 7 22:14:51.735750810 (5h48m5.322807052s) [seg_pos={0 0}, upload=5955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:14:51.735825741 (5h48m5.322881982s) [upload=5955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:14:51.735831010 (5h48m5.322887252s) [upload=5955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:28.158244845 (5h48m41.745301096s) [upload=5955]: Done uploading segments... Aug 7 22:15:28.158274540 (5h48m41.74533079s) [upload=5955]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:28.250312080 (5h48m41.837368322s) [upload=5969]: Starting upload Aug 7 22:15:28.250412248 (5h48m41.837468498s) [upload=5969]: Done starting upload Aug 7 22:15:28.250539435 (5h48m41.837595676s) [upload=5969]: Uploading segments... Aug 7 22:15:28.250814920 (5h48m41.83787116s) [upload=5969]: Got next segment. Inline: false Aug 7 22:15:28.250863259 (5h48m41.8379195s) [upload=5969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:28.250871685 (5h48m41.837927936s) [upload=5969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:58.237095607 (5h49m11.824151818s) [upload=5969]: Next segment err: context deadline exceeded Aug 7 22:15:58.237308174 (5h49m11.824364395s) [upload=5969]: Done uploading segments... Aug 7 22:15:58.237357947 (5h49m11.824414157s) [upload=5969]: Upload finished. err: context deadline exceeded Duration: 30.000686187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.X..................................XXXXXXX] duration: 30.000481836s FAIL (2025-08-07 22:15:55 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000500641s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX...X.....X..............................XXXXXX] duration: 30.000652737s FAIL (2025-08-07 22:15:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000669788s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XX.........................................XXXXX] duration: 30.00249005s FAIL (2025-08-07 22:15:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002505859s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX....................................XXXX] duration: 1m0.000574392s FAIL (2025-08-07 22:15:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00058377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXX.............................XXXXXXX] duration: 30.0003421s FAIL (2025-08-07 22:15:37 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Aug 7 22:08:48.176876088 (5h42m1.763932329s) [upload=5859]: Upload finished. err: <nil> Aug 7 22:09:04.330665274 (5h42m17.917721525s) [upload=5866]: Starting upload Aug 7 22:09:04.330750753 (5h42m17.917806994s) [upload=5866]: Done starting upload Aug 7 22:09:04.330821295 (5h42m17.917877535s) [upload=5866]: Uploading segments... Aug 7 22:09:04.330908929 (5h42m17.91796521s) [upload=5866]: Got next segment. Inline: true Aug 7 22:09:04.330923075 (5h42m17.917979317s) [upload=5866]: Waiting for error group managing segments... Aug 7 22:09:04.330991012 (5h42m17.918047253s) [upload=5866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:04.330999468 (5h42m17.918055718s) [upload=5866]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:04.331005178 (5h42m17.91806143s) [upload=5866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:04.636340599 (5h42m18.22339684s) [upload=5866]: Done uploading segments... Aug 7 22:09:04.636356889 (5h42m18.223413121s) [upload=5866]: Upload finished. err: <nil> Aug 7 22:09:43.052830170 (5h42m56.639886381s) [upload=5877]: Starting upload Aug 7 22:09:43.052883700 (5h42m56.639939921s) [upload=5877]: Done starting upload Aug 7 22:09:43.052934405 (5h42m56.639990626s) [upload=5877]: Uploading segments... Aug 7 22:09:43.052967446 (5h42m56.640023657s) [upload=5877]: Got next segment. Inline: true Aug 7 22:09:43.052971313 (5h42m56.640027534s) [upload=5877]: Waiting for error group managing segments... Aug 7 22:09:43.053000388 (5h42m56.640056609s) [upload=5877]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:43.053004064 (5h42m56.640060276s) [upload=5877]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:43.053006990 (5h42m56.640063211s) [upload=5877]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:43.127325938 (5h42m56.714382178s) [upload=5877]: Done uploading segments... Aug 7 22:09:43.127344483 (5h42m56.714400723s) [upload=5877]: Upload finished. err: <nil> Aug 7 22:10:15.358455928 (5h43m28.945512149s) [upload=5888]: Starting upload Aug 7 22:10:15.358522273 (5h43m28.945578494s) [upload=5888]: Done starting upload Aug 7 22:10:15.358588375 (5h43m28.945644586s) [upload=5888]: Uploading segments... Aug 7 22:10:15.358619654 (5h43m28.945675866s) [upload=5888]: Got next segment. Inline: true Aug 7 22:10:15.358623641 (5h43m28.945679853s) [upload=5888]: Waiting for error group managing segments... Aug 7 22:10:15.358657604 (5h43m28.945713826s) [upload=5888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:15.358661782 (5h43m28.945717993s) [upload=5888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:15.358665169 (5h43m28.94572139s) [upload=5888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:15.541784918 (5h43m29.128841169s) [upload=5888]: Done uploading segments... Aug 7 22:10:15.541804064 (5h43m29.128860304s) [upload=5888]: Upload finished. err: <nil> Aug 7 22:11:10.710381989 (5h44m24.29743823s) [upload=5902]: Starting upload Aug 7 22:11:10.710469673 (5h44m24.297525913s) [upload=5902]: Done starting upload Aug 7 22:11:10.710537730 (5h44m24.297593971s) [upload=5902]: Uploading segments... Aug 7 22:11:10.710600456 (5h44m24.297656698s) [upload=5902]: Got next segment. Inline: true Aug 7 22:11:10.710607830 (5h44m24.297664072s) [upload=5902]: Waiting for error group managing segments... Aug 7 22:11:10.710658876 (5h44m24.297715118s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:10.710666100 (5h44m24.297722341s) [upload=5902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:10.710670057 (5h44m24.297726299s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:40.695011486 (5h44m54.282067737s) [upload=5902]: Done uploading segments... Aug 7 22:11:40.695031373 (5h44m54.282087624s) [upload=5902]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:11:40.801935294 (5h44m54.388991515s) [upload=5908]: Starting upload Aug 7 22:11:40.802009452 (5h44m54.389065673s) [upload=5908]: Done starting upload Aug 7 22:11:40.802061539 (5h44m54.38911776s) [upload=5908]: Uploading segments... Aug 7 22:11:40.802108137 (5h44m54.389164358s) [upload=5908]: Got next segment. Inline: true Aug 7 22:11:40.802112685 (5h44m54.389168896s) [upload=5908]: Waiting for error group managing segments... Aug 7 22:11:40.802146128 (5h44m54.389202338s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.802150235 (5h44m54.389206456s) [upload=5908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:40.802172947 (5h44m54.389229168s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.785746890 (5h45m24.372803111s) [upload=5908]: Done uploading segments... Aug 7 22:12:10.785765736 (5h45m24.372821946s) [upload=5908]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:34.227606352 (5h45m47.814662563s) [upload=5920]: Starting upload Aug 7 22:12:34.227679809 (5h45m47.81473602s) [upload=5920]: Done starting upload Aug 7 22:12:34.227726427 (5h45m47.814782637s) [upload=5920]: Uploading segments... Aug 7 22:12:34.227755791 (5h45m47.814812012s) [upload=5920]: Got next segment. Inline: true Aug 7 22:12:34.227759368 (5h45m47.814815579s) [upload=5920]: Waiting for error group managing segments... Aug 7 22:12:34.227789123 (5h45m47.814845344s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:34.227792860 (5h45m47.814849081s) [upload=5920]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:34.227795325 (5h45m47.814851546s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:04.214429951 (5h46m17.801486172s) [upload=5920]: Done uploading segments... Aug 7 22:13:04.214444368 (5h46m17.801500579s) [upload=5920]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:21.693003516 (5h46m35.280059727s) [upload=5934]: Starting upload Aug 7 22:13:21.693054191 (5h46m35.280110402s) [upload=5934]: Done starting upload Aug 7 22:13:21.693097442 (5h46m35.280153663s) [upload=5934]: Uploading segments... Aug 7 22:13:21.693124161 (5h46m35.280180382s) [upload=5934]: Got next segment. Inline: true Aug 7 22:13:21.693127627 (5h46m35.280183848s) [upload=5934]: Waiting for error group managing segments... Aug 7 22:13:21.693172341 (5h46m35.280228552s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:21.693176398 (5h46m35.28023262s) [upload=5934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:21.693178873 (5h46m35.280235085s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:51.677769243 (5h47m5.264825484s) [upload=5934]: Done uploading segments... Aug 7 22:13:51.677786175 (5h47m5.264842416s) [upload=5934]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:51.763566605 (5h47m5.350622826s) [upload=5942]: Starting upload Aug 7 22:13:51.763615116 (5h47m5.350671337s) [upload=5942]: Done starting upload Aug 7 22:13:51.763654910 (5h47m5.350711131s) [upload=5942]: Uploading segments... Aug 7 22:13:51.763683413 (5h47m5.350739634s) [upload=5942]: Got next segment. Inline: true Aug 7 22:13:51.763687000 (5h47m5.350743211s) [upload=5942]: Waiting for error group managing segments... Aug 7 22:13:51.763718620 (5h47m5.350774841s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.763722296 (5h47m5.350778517s) [upload=5942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:51.763724580 (5h47m5.350780801s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:21.752690510 (5h47m35.339746731s) [upload=5942]: Done uploading segments... Aug 7 22:14:21.752712410 (5h47m35.339768631s) [upload=5942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:21.832761353 (5h47m35.419817564s) [upload=5952]: Starting upload Aug 7 22:14:21.832808701 (5h47m35.419864913s) [upload=5952]: Done starting upload Aug 7 22:14:21.832873814 (5h47m35.419930035s) [upload=5952]: Uploading segments... Aug 7 22:14:21.832905782 (5h47m35.419961993s) [upload=5952]: Got next segment. Inline: true Aug 7 22:14:21.832909440 (5h47m35.419965661s) [upload=5952]: Waiting for error group managing segments... Aug 7 22:14:21.832940607 (5h47m35.419996828s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:21.832944185 (5h47m35.420000406s) [upload=5952]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:21.832946779 (5h47m35.420003s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:51.821960613 (5h48m5.409016855s) [upload=5952]: Done uploading segments... Aug 7 22:14:51.821995769 (5h48m5.40905202s) [upload=5952]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:07.153250299 (5h48m20.74030654s) [upload=5963]: Starting upload Aug 7 22:15:07.153363971 (5h48m20.740420213s) [upload=5963]: Done starting upload Aug 7 22:15:07.153422571 (5h48m20.740478811s) [upload=5963]: Uploading segments... Aug 7 22:15:07.153473596 (5h48m20.740529837s) [upload=5963]: Got next segment. Inline: true Aug 7 22:15:07.153480569 (5h48m20.74053681s) [upload=5963]: Waiting for error group managing segments... Aug 7 22:15:07.153529239 (5h48m20.740585481s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:07.153535742 (5h48m20.740591973s) [upload=5963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:07.153539228 (5h48m20.740595469s) [upload=5963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:37.139229671 (5h48m50.726285892s) [upload=5963]: Done uploading segments... Aug 7 22:15:37.139243938 (5h48m50.726300148s) [upload=5963]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000360955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXX..X........................................XXXXX] duration: 48.667702187s FAIL (2025-08-07 22:15:28 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pb.BatchRequestItem_ObjectBegin Aug 7 22:14:09.430302826 (5h47m23.017359037s) [upload=5948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:39.419494306 (5h47m53.006550557s) [upload=5948]: Next segment err: context deadline exceeded Aug 7 22:14:39.419690903 (5h47m53.006747145s) [upload=5948]: Done uploading segments... Aug 7 22:14:39.419719496 (5h47m53.006775737s) [upload=5948]: Upload finished. err: context deadline exceeded Aug 7 22:14:39.504854221 (5h47m53.091910472s) [upload=5955]: Starting upload Aug 7 22:14:39.504946442 (5h47m53.092002684s) [upload=5955]: Done starting upload Aug 7 22:14:39.505044656 (5h47m53.092100897s) [upload=5955]: Uploading segments... Aug 7 22:14:39.505363441 (5h47m53.092419682s) [upload=5955]: Got next segment. Inline: false Aug 7 22:14:39.505499695 (5h47m53.092555936s) [upload=5955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:39.505507009 (5h47m53.09256325s) [upload=5955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:51.424409681 (5h48m5.011465921s) [seg_pos={0 0}, upload=5955]: Begin upload segment... Aug 7 22:14:51.424474331 (5h48m5.011530572s) [seg_pos={0 0}, upload=5955]: Done begin upload segment. Aug 7 22:14:51.424503616 (5h48m5.011559867s) [noise=true , address=104.36.82.15:4257 , piece_id=C6V3O2TFBIF5W2E4AJA6BIRK7CJOQUJ73H5SVYPB5KYXUZ3HN5PA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.424521709 (5h48m5.011577921s) [noise=true , address=45.91.116.14:4189 , piece_id=65YS2EZ7TJFMO2RHD6F2SGUN2CXHJ47AI3NOX4G45GCKR663LXTQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.424865801 (5h48m5.011922023s) [noise=true , address=45.91.116.12:4226 , piece_id=2YPECU45KNEFQEABDWQNUY6DTGZZWUFE5YK767DQSZKXW6AHJBDA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.424976539 (5h48m5.01203278s) [noise=true , address=45.91.116.12:4227 , piece_id=PT72AOCQEVQJ2ZSMYVM4FHBQSKMJDSGMKRAHRRS6W4QVTWNIYEAQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425100399 (5h48m5.01215661s) [noise=true , address=77.90.31.13:4626 , piece_id=4WWEMP7VRA72YTDSGUSQHVACDKAPJNH5JO3XJQQDS5A3VY7HLENA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425376945 (5h48m5.012433187s) [noise=true , address=104.36.82.16:4294 , piece_id=GM6LYNFPDJVGPTJA7ZIQYZFEIMNSCQYRSNO4TPWYZJJ6S6N63HNA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.425848076 (5h48m5.012904316s) [noise=true , address=45.91.116.14:4190 , piece_id=D64JHW36T5LGBYG6IDZVYQD4GZWCYQI55LR4QOHOPCDRTUJJHMYA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.426300299 (5h48m5.013356541s) [noise=true , address=77.90.31.15:4697 , piece_id=XDCICPLIEUA3HBHPTL5YKQ5MITBASANALKMAXLEAATHSTA23HWBQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.426661613 (5h48m5.013717855s) [noise=true , address=45.91.116.10:4082 , piece_id=MCI5J5AEY44AJPHKEVSTVZGDBHRNMG4R5GCIASZ2OXX4F2JMIKMQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.427149574 (5h48m5.014205816s) [noise=true , address=77.90.31.17:4772 , piece_id=WLFZC5FRPMYYQDN4JZVPYDCNHDMVMNL2AKV4F5UKQQ7JM4JYXDFA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.427626695 (5h48m5.014682937s) [noise=true , address=77.90.31.16:4735 , piece_id=HQ2BPD52ZQXRH5YHMX7JGS2JJ5I5BQALG327UMWAYUW7W245N4QQ, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428103996 (5h48m5.015160247s) [noise=true , address=104.36.82.6:4873 , piece_id=D5QPVOWRUSHU5HITFOBHNYS3J7REL5XOROWGLJC4EAHXNH7EJ73A, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428503051 (5h48m5.015559292s) [noise=true , address=104.36.82.7:4912 , piece_id=QBFXRWNLJKWEOJ36KB6AJMOBZ64BW5NT4UZINBVGQHQZDLFXP5YA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.428861140 (5h48m5.01591739s) [noise=true , address=45.91.116.5:4299 , piece_id=KP6S3WEZGKH2V5OE34LLHHSNQ5XPRKSKFNK66DVOTMWOWBJCOZQA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.429259873 (5h48m5.016316115s) [noise=true , address=45.91.116.9:4261 , piece_id=JNPPGNLOROFBT27FHNEU6HQY77NEPFQBZOHJZRDAPCUO3UOCKR2Q, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.429662896 (5h48m5.016719138s) [noise=true , address=77.90.31.17:4770 , piece_id=EGU57FLC2PJOV3AXAK2NDGYGHISFH3L3GFK5ZMWUM45MG3RUUOOA, seg_pos={0 0}, upload=5955]: Uploading piece... Aug 7 22:14:51.550946177 (5h48m5.138002409s) [upload=5955]: Next returned nil segment Aug 7 22:14:51.550961846 (5h48m5.138018078s) [upload=5955]: Waiting for error group managing segments... Aug 7 22:14:51.559757896 (5h48m5.146814117s) [noise=true , address=45.91.116.14:4189 , piece_id=65YS2EZ7TJFMO2RHD6F2SGUN2CXHJ47AI3NOX4G45GCKR663LXTQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561017878 (5h48m5.148074129s) [noise=true , address=45.91.116.12:4227 , piece_id=PT72AOCQEVQJ2ZSMYVM4FHBQSKMJDSGMKRAHRRS6W4QVTWNIYEAQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561713717 (5h48m5.148769938s) [noise=true , address=45.91.116.10:4082 , piece_id=MCI5J5AEY44AJPHKEVSTVZGDBHRNMG4R5GCIASZ2OXX4F2JMIKMQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.561816609 (5h48m5.14887284s) [noise=true , address=45.91.116.5:4299 , piece_id=KP6S3WEZGKH2V5OE34LLHHSNQ5XPRKSKFNK66DVOTMWOWBJCOZQA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.562321571 (5h48m5.149377812s) [noise=true , address=45.91.116.9:4261 , piece_id=JNPPGNLOROFBT27FHNEU6HQY77NEPFQBZOHJZRDAPCUO3UOCKR2Q, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.563667614 (5h48m5.150723845s) [noise=true , address=77.90.31.15:4697 , piece_id=XDCICPLIEUA3HBHPTL5YKQ5MITBASANALKMAXLEAATHSTA23HWBQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.565443659 (5h48m5.15249991s) [noise=true , address=104.36.82.15:4257 , piece_id=C6V3O2TFBIF5W2E4AJA6BIRK7CJOQUJ73H5SVYPB5KYXUZ3HN5PA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.566941915 (5h48m5.153998147s) [noise=true , address=104.36.82.7:4912 , piece_id=QBFXRWNLJKWEOJ36KB6AJMOBZ64BW5NT4UZINBVGQHQZDLFXP5YA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.567819224 (5h48m5.154875466s) [noise=true , address=45.91.116.14:4190 , piece_id=D64JHW36T5LGBYG6IDZVYQD4GZWCYQI55LR4QOHOPCDRTUJJHMYA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.568503540 (5h48m5.155559782s) [noise=true , address=104.36.82.6:4873 , piece_id=D5QPVOWRUSHU5HITFOBHNYS3J7REL5XOROWGLJC4EAHXNH7EJ73A, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.576217691 (5h48m5.163273931s) [noise=true , address=45.91.116.12:4226 , piece_id=2YPECU45KNEFQEABDWQNUY6DTGZZWUFE5YK767DQSZKXW6AHJBDA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.578272647 (5h48m5.165328888s) [noise=true , address=77.90.31.17:4770 , piece_id=EGU57FLC2PJOV3AXAK2NDGYGHISFH3L3GFK5ZMWUM45MG3RUUOOA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.735239817 (5h48m5.322296068s) [noise=true , address=104.36.82.16:4294 , piece_id=GM6LYNFPDJVGPTJA7ZIQYZFEIMNSCQYRSNO4TPWYZJJ6S6N63HNA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: <nil> Aug 7 22:14:51.735279151 (5h48m5.322335392s) [seg_pos={0 0}, upload=5955]: Segment reached optimal threshold of 13 pieces. Aug 7 22:14:51.735633281 (5h48m5.322689532s) [noise=true , address=77.90.31.17:4772 , piece_id=WLFZC5FRPMYYQDN4JZVPYDCNHDMVMNL2AKV4F5UKQQ7JM4JYXDFA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection; write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:62798->77.90.31.17:4772: use of closed network connection Aug 7 22:14:51.735644341 (5h48m5.322700583s) [noise=true , address=77.90.31.16:4735 , piece_id=HQ2BPD52ZQXRH5YHMX7JGS2JJ5I5BQALG327UMWAYUW7W245N4QQ, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63636->77.90.31.16:4735: use of closed network connection Aug 7 22:14:51.735698282 (5h48m5.322754524s) [noise=true , address=77.90.31.13:4626 , piece_id=4WWEMP7VRA72YTDSGUSQHVACDKAPJNH5JO3XJQQDS5A3VY7HLENA, seg_pos={0 0}, upload=5955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection; write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:43034->77.90.31.13:4626: use of closed network connection Aug 7 22:14:51.735750810 (5h48m5.322807052s) [seg_pos={0 0}, upload=5955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:14:51.735825741 (5h48m5.322881982s) [upload=5955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:14:51.735831010 (5h48m5.322887252s) [upload=5955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:28.158244845 (5h48m41.745301096s) [upload=5955]: Done uploading segments... Aug 7 22:15:28.158274540 (5h48m41.74533079s) [upload=5955]: Upload finished. err: metaclient: context deadline exceeded Duration: 48.667722516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XX......X.............................XXXX] duration: 1m0.001389705s FAIL (2025-08-07 22:15:22 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== .82.4:4827 , piece_id=QP6JUIBNRV7YFX433AN5PLRGTRPQQ2XC52HF7EYQSGF3FMAXM4JA, seg_pos={0 0}, upload=5882]: Uploading piece... Aug 7 22:10:00.328911351 (5h43m13.915967563s) [upload=5882]: Next returned nil segment Aug 7 22:10:00.328920107 (5h43m13.915976319s) [upload=5882]: Waiting for error group managing segments... Aug 7 22:10:00.356440428 (5h43m13.943496669s) [noise=true , address=104.36.82.62:4420 , piece_id=B4KTTAN5WEAXLXS3O5RPYU4AYJ6YJOGL5PDPDDCCOLXUU63RK3GQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.362225756 (5h43m13.949281997s) [noise=true , address=104.36.82.60:4367 , piece_id=5F72CGA55FJGB5MRV42IBWWBOJBPQL45AQRUALONFHSLEQAQVDCA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.365929300 (5h43m13.952985542s) [noise=true , address=104.36.82.52:4114 , piece_id=6UXFIJCYQO7TEZDFZXS3M53PGM7LOPDNB3ARCXNGCNKOJQGDEY4Q, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.367401928 (5h43m13.954458149s) [noise=true , address=104.36.82.59:4323 , piece_id=RDLGI7JXFHQINYTMPXTQLIREY6SOQJJ4YIJSCUG3UEZWXLWNGMNQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.368760244 (5h43m13.955816485s) [noise=true , address=104.36.82.59:4309 , piece_id=YYKRP337BNKDZYEEK5P7MDA62ELU5VGRSRB6ML76DKG6XYQS5QGQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.390444030 (5h43m13.977500241s) [noise=true , address=104.36.82.55:4212 , piece_id=R44VQAJ47VZZ622QQSXSS6ZLYJVQOYHVJFBK5XLLPHDLTPZG3OQA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.391147313 (5h43m13.978203534s) [noise=true , address=104.36.82.15:4268 , piece_id=EFSX37BH3XS3HWGET5VD3RYRGJWJK4PDGXG45IVJKB7SOSVP7YZA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.399754149 (5h43m13.98681037s) [noise=true , address=104.36.82.9:4011 , piece_id=UT6I3AJEWBCSPWUJPGJK3BUB6SZMFAZ5EQBAQ56Q65BFAAIIROIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.460854191 (5h43m14.047910432s) [noise=true , address=104.36.82.42:4799 , piece_id=LXQBRMRAEKV5OY2L5WGUW4E43RMLFBHYZRTZ2IUGMUDENRDBQLFQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.470345858 (5h43m14.0574021s) [noise=true , address=104.36.82.43:4846 , piece_id=3HN5X5L6FZOM2LY6UG6FUPFQSJGVCISOHFDI4T6IIW4XRCV7EKCQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.507622690 (5h43m14.094678901s) [noise=true , address=104.36.82.45:4913 , piece_id=FN5X5ZS7L7HRERGTFSBHDONB5V57E2AE5RG55CEAS6WPI6L7VWFQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.534357043 (5h43m14.121413254s) [noise=true , address=104.36.82.58:4279 , piece_id=Z2FXWDDE6Q4S5P73GMZFI6RM5LBBRWMDLKH6XK7MF3IAUWZC6PIA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.552361511 (5h43m14.139417732s) [noise=true , address=104.36.82.4:4827 , piece_id=QP6JUIBNRV7YFX433AN5PLRGTRPQQ2XC52HF7EYQSGF3FMAXM4JA, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.596411237 (5h43m14.183467448s) [noise=true , address=104.36.82.56:4241 , piece_id=CZNEIWPUR5RWWSRD42ZZV76SKKYQUL22K5OSMRABZ3JLNWPOQQPQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: <nil> Aug 7 22:10:00.596432907 (5h43m14.183489128s) [seg_pos={0 0}, upload=5882]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:00.596673115 (5h43m14.183729336s) [noise=true , address=104.36.82.13:4197 , piece_id=YXUCKQYQYWVKLTOR6AZHGS5LANJQESFHAGE6F2ZA272CZDT2BUIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM): protocol: write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection; write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:26338->104.36.82.13:4197: use of closed network connection Aug 7 22:10:00.596697370 (5h43m14.183753591s) [noise=true , address=104.36.82.16:4327 , piece_id=GRKTSALMEB73JLA74SJ6ET75F3JY4PXVTYOGTANJBS3VOETOVBIQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection; write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:47746->104.36.82.16:4327: use of closed network connection Aug 7 22:10:00.596712248 (5h43m14.183768469s) [noise=true , address=79.127.205.234:4600 , piece_id=LP74NCGX4H2E4U5UHJATTAARN6E4O4MVSVP2ED2N67LXJQX7DPAQ, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kxaNeRZqxza8AxSaF4PvBkmKoUY2NvficDEBRaXfDnq1HSgJ1): protocol: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection; write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:17592->79.127.205.234:4600: use of closed network connection Aug 7 22:10:00.596771750 (5h43m14.183827971s) [noise=true , address=104.36.82.37:4685 , piece_id=4Q4WHVJWI4VXAEZEPTS355RHEBI37ME5S3MDUI5KM45LVDZ3GZ3A, seg_pos={0 0}, upload=5882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MhzeNvbKYhZVnhQW8oY5BeGQDuyNtmi25Xhgzrej4bCyoD9VM): protocol: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:31876->104.36.82.37:4685: use of closed network connection Aug 7 22:10:00.596795714 (5h43m14.183851935s) [seg_pos={0 0}, upload=5882]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:00.596866456 (5h43m14.183922677s) [upload=5882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:00.596873089 (5h43m14.18392931s) [upload=5882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:00.660018838 (5h43m14.247075089s) [upload=5882]: Done uploading segments... Aug 7 22:10:00.660039907 (5h43m14.247096149s) [upload=5882]: Upload finished. err: <nil> Aug 7 22:11:14.838405386 (5h44m28.425461627s) [upload=5904]: Starting upload Aug 7 22:11:14.838502537 (5h44m28.425558788s) [upload=5904]: Done starting upload Aug 7 22:11:14.838673375 (5h44m28.425729627s) [upload=5904]: Uploading segments... Aug 7 22:11:14.838929564 (5h44m28.425985815s) [upload=5904]: Got next segment. Inline: false Aug 7 22:11:14.838997871 (5h44m28.426054112s) [upload=5904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:14.839003762 (5h44m28.426060003s) [upload=5904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:14.826003908 (5h45m28.413060119s) [upload=5904]: Next segment err: context deadline exceeded Aug 7 22:12:14.826061727 (5h45m28.413117948s) [upload=5904]: Done uploading segments... Aug 7 22:12:14.826077997 (5h45m28.413134218s) [upload=5904]: Upload finished. err: context deadline exceeded Aug 7 22:12:14.896180564 (5h45m28.483236775s) [upload=5918]: Starting upload Aug 7 22:12:14.896231629 (5h45m28.483287841s) [upload=5918]: Done starting upload Aug 7 22:12:14.896288415 (5h45m28.483344636s) [upload=5918]: Uploading segments... Aug 7 22:12:14.896472659 (5h45m28.48352888s) [upload=5918]: Got next segment. Inline: false Aug 7 22:12:14.896503706 (5h45m28.483559918s) [upload=5918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:14.896507904 (5h45m28.483564126s) [upload=5918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:14.884243492 (5h46m28.471299734s) [upload=5918]: Next segment err: context deadline exceeded Aug 7 22:13:14.884316208 (5h46m28.471372449s) [upload=5918]: Done uploading segments... Aug 7 22:13:14.884334913 (5h46m28.471391154s) [upload=5918]: Upload finished. err: context deadline exceeded Aug 7 22:13:15.003888637 (5h46m28.590944898s) [upload=5932]: Starting upload Aug 7 22:13:15.004039338 (5h46m28.591095609s) [upload=5932]: Done starting upload Aug 7 22:13:15.004748231 (5h46m28.591804483s) [upload=5932]: Uploading segments... Aug 7 22:13:15.004838500 (5h46m28.591894741s) [upload=5932]: Got next segment. Inline: false Aug 7 22:13:15.004895547 (5h46m28.591951788s) [upload=5932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:15.004902439 (5h46m28.591958681s) [upload=5932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:14.988056084 (5h47m28.575112325s) [upload=5932]: Next segment err: context deadline exceeded Aug 7 22:14:14.988128109 (5h47m28.575184351s) [upload=5932]: Done uploading segments... Aug 7 22:14:14.988155329 (5h47m28.57521157s) [upload=5932]: Upload finished. err: context deadline exceeded Aug 7 22:14:22.127816864 (5h47m35.714873075s) [upload=5953]: Starting upload Aug 7 22:14:22.127868040 (5h47m35.714924261s) [upload=5953]: Done starting upload Aug 7 22:14:22.127944151 (5h47m35.715000372s) [upload=5953]: Uploading segments... Aug 7 22:14:22.128112817 (5h47m35.715169038s) [upload=5953]: Got next segment. Inline: false Aug 7 22:14:22.128145027 (5h47m35.715201248s) [upload=5953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:22.128148994 (5h47m35.715205205s) [upload=5953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:22.111818521 (5h48m35.698874742s) [upload=5953]: Next segment err: context deadline exceeded Aug 7 22:15:22.111891768 (5h48m35.698947999s) [upload=5953]: Done uploading segments... Aug 7 22:15:22.111945839 (5h48m35.69900206s) [upload=5953]: Upload finished. err: context deadline exceeded Duration: 1m0.001409151s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX...X.....X..............................XXXXX] duration: 30.00054321s FAIL (2025-08-07 22:15:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000556525s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.X...X...X..................................................] duration: 241.501945ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:15:05 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 241.528915ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 333.351868ms FAIL (2025-08-07 22:15:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== ine Aug 7 21:59:56.515336538 (5h33m10.10239275s) [upload=5708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:59:56.574152557 (5h33m10.161208778s) [upload=5708]: Done uploading segments... Aug 7 21:59:56.574171943 (5h33m10.161228164s) [upload=5708]: Upload finished. err: <nil> Aug 7 22:00:36.611785100 (5h33m50.19884134s) [upload=5716]: Starting upload Aug 7 22:00:36.611856203 (5h33m50.198912444s) [upload=5716]: Done starting upload Aug 7 22:00:36.611959786 (5h33m50.199016027s) [upload=5716]: Uploading segments... Aug 7 22:00:36.612003668 (5h33m50.199059918s) [upload=5716]: Got next segment. Inline: true Aug 7 22:00:36.612009668 (5h33m50.19906591s) [upload=5716]: Waiting for error group managing segments... Aug 7 22:00:36.612057077 (5h33m50.199113319s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:00:36.612082536 (5h33m50.199138776s) [upload=5716]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:00:36.612086222 (5h33m50.199142473s) [upload=5716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:00:36.884781084 (5h33m50.471837294s) [upload=5716]: Done uploading segments... Aug 7 22:00:36.884795611 (5h33m50.471851832s) [upload=5716]: Upload finished. err: <nil> Aug 7 22:01:47.555449092 (5h35m1.142505313s) [upload=5736]: Starting upload Aug 7 22:01:47.555491781 (5h35m1.142548002s) [upload=5736]: Done starting upload Aug 7 22:01:47.555562123 (5h35m1.142618344s) [upload=5736]: Uploading segments... Aug 7 22:01:47.555595826 (5h35m1.142652047s) [upload=5736]: Got next segment. Inline: true Aug 7 22:01:47.555599363 (5h35m1.142655584s) [upload=5736]: Waiting for error group managing segments... Aug 7 22:01:47.555631602 (5h35m1.142687823s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:01:47.555635390 (5h35m1.142691611s) [upload=5736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:01:47.555637894 (5h35m1.142694115s) [upload=5736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:01:47.686110188 (5h35m1.273166429s) [upload=5736]: Done uploading segments... Aug 7 22:01:47.686151165 (5h35m1.273207406s) [upload=5736]: Upload finished. err: <nil> Aug 7 22:02:22.678566055 (5h35m36.265622267s) [upload=5746]: Starting upload Aug 7 22:02:22.678633091 (5h35m36.265689303s) [upload=5746]: Done starting upload Aug 7 22:02:22.678684407 (5h35m36.265740617s) [upload=5746]: Uploading segments... Aug 7 22:02:22.678721757 (5h35m36.265777967s) [upload=5746]: Got next segment. Inline: true Aug 7 22:02:22.678725724 (5h35m36.265781935s) [upload=5746]: Waiting for error group managing segments... Aug 7 22:02:22.678759457 (5h35m36.265815668s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:02:22.678764947 (5h35m36.265821168s) [upload=5746]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:02:22.678768565 (5h35m36.265824826s) [upload=5746]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:02:22.971349697 (5h35m36.558405918s) [upload=5746]: Done uploading segments... Aug 7 22:02:22.971373742 (5h35m36.558429963s) [upload=5746]: Upload finished. err: <nil> Aug 7 22:03:08.907032448 (5h36m22.494088689s) [upload=5757]: Starting upload Aug 7 22:03:08.907091528 (5h36m22.494147769s) [upload=5757]: Done starting upload Aug 7 22:03:08.907172368 (5h36m22.4942286s) [upload=5757]: Uploading segments... Aug 7 22:03:08.907209458 (5h36m22.494265699s) [upload=5757]: Got next segment. Inline: true Aug 7 22:03:08.907214758 (5h36m22.494271s) [upload=5757]: Waiting for error group managing segments... Aug 7 22:03:08.907255936 (5h36m22.494312176s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:03:08.907261175 (5h36m22.494317416s) [upload=5757]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:03:08.907265904 (5h36m22.494322135s) [upload=5757]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:03:09.004207779 (5h36m22.59126404s) [upload=5757]: Done uploading segments... Aug 7 22:03:09.004232375 (5h36m22.591288627s) [upload=5757]: Upload finished. err: <nil> Aug 7 22:04:02.463086309 (5h37m16.050142521s) [upload=5775]: Starting upload Aug 7 22:04:02.463128347 (5h37m16.050184568s) [upload=5775]: Done starting upload Aug 7 22:04:02.463197818 (5h37m16.050254029s) [upload=5775]: Uploading segments... Aug 7 22:04:02.463223595 (5h37m16.050279816s) [upload=5775]: Got next segment. Inline: true Aug 7 22:04:02.463227652 (5h37m16.050283873s) [upload=5775]: Waiting for error group managing segments... Aug 7 22:04:02.463258120 (5h37m16.050314331s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:04:02.463261796 (5h37m16.050318008s) [upload=5775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:04:02.463264733 (5h37m16.050320954s) [upload=5775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:02.530547234 (5h37m16.117603445s) [upload=5775]: Done uploading segments... Aug 7 22:04:02.530562512 (5h37m16.117618733s) [upload=5775]: Upload finished. err: <nil> Aug 7 22:05:10.744065032 (5h38m24.331121272s) [upload=5796]: Starting upload Aug 7 22:05:10.744195716 (5h38m24.331251957s) [upload=5796]: Done starting upload Aug 7 22:05:10.744327762 (5h38m24.331384002s) [upload=5796]: Uploading segments... Aug 7 22:05:10.744369549 (5h38m24.331425791s) [upload=5796]: Got next segment. Inline: true Aug 7 22:05:10.744388655 (5h38m24.331444897s) [upload=5796]: Waiting for error group managing segments... Aug 7 22:05:10.744449028 (5h38m24.331505259s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:10.744454398 (5h38m24.331510629s) [upload=5796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:10.744457633 (5h38m24.331513875s) [upload=5796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:10.852991578 (5h38m24.440047789s) [upload=5796]: Done uploading segments... Aug 7 22:05:10.853004571 (5h38m24.440060782s) [upload=5796]: Upload finished. err: <nil> Aug 7 22:05:45.060720865 (5h38m58.647777086s) [upload=5806]: Starting upload Aug 7 22:05:45.060780046 (5h38m58.647836267s) [upload=5806]: Done starting upload Aug 7 22:05:45.060842121 (5h38m58.647898333s) [upload=5806]: Uploading segments... Aug 7 22:05:45.060876836 (5h38m58.647933057s) [upload=5806]: Got next segment. Inline: true Aug 7 22:05:45.060882878 (5h38m58.647939099s) [upload=5806]: Waiting for error group managing segments... Aug 7 22:05:45.060927901 (5h38m58.647984112s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:45.060933843 (5h38m58.647990064s) [upload=5806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:45.060936187 (5h38m58.647992408s) [upload=5806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:45.328399453 (5h38m58.915455674s) [upload=5806]: Done uploading segments... Aug 7 22:05:45.328433367 (5h38m58.915489588s) [upload=5806]: Upload finished. err: <nil> Aug 7 22:06:12.931860875 (5h39m26.518917096s) [upload=5815]: Starting upload Aug 7 22:06:12.931905979 (5h39m26.51896219s) [upload=5815]: Done starting upload Aug 7 22:06:12.931964518 (5h39m26.51902073s) [upload=5815]: Uploading segments... Aug 7 22:06:12.931990617 (5h39m26.519046827s) [upload=5815]: Got next segment. Inline: true Aug 7 22:06:12.931994224 (5h39m26.519050434s) [upload=5815]: Waiting for error group managing segments... Aug 7 22:06:12.932024821 (5h39m26.519081042s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:12.932028408 (5h39m26.519084618s) [upload=5815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:12.932030893 (5h39m26.519087103s) [upload=5815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:13.014995076 (5h39m26.602051317s) [upload=5815]: Done uploading segments... Aug 7 22:06:13.015014863 (5h39m26.602071114s) [upload=5815]: Upload finished. err: <nil> Aug 7 22:07:16.916734820 (5h40m30.503791062s) [upload=5832]: Starting upload Aug 7 22:07:16.916800323 (5h40m30.503856563s) [upload=5832]: Done starting upload Aug 7 22:07:16.916864883 (5h40m30.503921123s) [upload=5832]: Uploading segments... Aug 7 22:07:16.916903304 (5h40m30.503959546s) [upload=5832]: Got next segment. Inline: true Aug 7 22:07:16.916908815 (5h40m30.503965056s) [upload=5832]: Waiting for error group managing segments... Aug 7 22:07:16.916952356 (5h40m30.504008598s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:16.916957657 (5h40m30.504013897s) [upload=5832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:16.916961754 (5h40m30.504017995s) [upload=5832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:16.988776316 (5h40m30.575832566s) [upload=5832]: Done uploading segments... Aug 7 22:07:16.988801573 (5h40m30.575857824s) [upload=5832]: Upload finished. err: <nil> Aug 7 22:08:29.650294124 (5h41m43.237350365s) [upload=5854]: Starting upload Aug 7 22:08:29.650357383 (5h41m43.237413623s) [upload=5854]: Done starting upload Aug 7 22:08:29.650418325 (5h41m43.237474567s) [upload=5854]: Uploading segments... Aug 7 22:08:29.650468490 (5h41m43.237524731s) [upload=5854]: Got next segment. Inline: true Aug 7 22:08:29.650474350 (5h41m43.237530592s) [upload=5854]: Waiting for error group managing segments... Aug 7 22:08:29.650515618 (5h41m43.237571859s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:29.650522029 (5h41m43.237578271s) [upload=5854]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:29.650525556 (5h41m43.237581797s) [upload=5854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:29.706715553 (5h41m43.293771793s) [upload=5854]: Done uploading segments... Aug 7 22:08:29.706731001 (5h41m43.293787243s) [upload=5854]: Upload finished. err: metaclient: Too Many Requests Duration:* 333.385791ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XX.........................................XXXX] duration: 30.001146419s FAIL (2025-08-07 22:15:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001167558s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.X.XX...............................XX] duration: 1m0.000503809s FAIL (2025-08-07 22:14:59 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [upload=5782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:04:30.993217718 (5h37m44.580273939s) [upload=5782]: Done uploading segments... Aug 7 22:04:30.993242384 (5h37m44.580298605s) [upload=5782]: Upload finished. err: <nil> Aug 7 22:05:13.839227971 (5h38m27.426284192s) [upload=5797]: Starting upload Aug 7 22:05:13.839278486 (5h38m27.426334707s) [upload=5797]: Done starting upload Aug 7 22:05:13.839330563 (5h38m27.426386774s) [upload=5797]: Uploading segments... Aug 7 22:05:13.839354638 (5h38m27.426410849s) [upload=5797]: Got next segment. Inline: true Aug 7 22:05:13.839358215 (5h38m27.426414436s) [upload=5797]: Waiting for error group managing segments... Aug 7 22:05:13.839391387 (5h38m27.426447608s) [upload=5797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:13.839395324 (5h38m27.426451535s) [upload=5797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:13.839397699 (5h38m27.42645392s) [upload=5797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:13.937726083 (5h38m27.524782293s) [upload=5797]: Done uploading segments... Aug 7 22:05:13.937747112 (5h38m27.524803323s) [upload=5797]: Upload finished. err: <nil> Aug 7 22:06:14.837051759 (5h39m28.42410798s) [upload=5816]: Starting upload Aug 7 22:06:14.837104066 (5h39m28.424160287s) [upload=5816]: Done starting upload Aug 7 22:06:14.837167315 (5h39m28.424223526s) [upload=5816]: Uploading segments... Aug 7 22:06:14.837196950 (5h39m28.42425316s) [upload=5816]: Got next segment. Inline: true Aug 7 22:06:14.837201157 (5h39m28.424257379s) [upload=5816]: Waiting for error group managing segments... Aug 7 22:06:14.837234911 (5h39m28.424291132s) [upload=5816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:14.837239529 (5h39m28.42429575s) [upload=5816]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:14.837241954 (5h39m28.424298175s) [upload=5816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:14.892558219 (5h39m28.47961446s) [upload=5816]: Done uploading segments... Aug 7 22:06:14.892577646 (5h39m28.479633886s) [upload=5816]: Upload finished. err: <nil> Aug 7 22:06:56.585938978 (5h40m10.172995189s) [upload=5826]: Starting upload Aug 7 22:06:56.585989272 (5h40m10.173045493s) [upload=5826]: Done starting upload Aug 7 22:06:56.586034606 (5h40m10.173090827s) [upload=5826]: Uploading segments... Aug 7 22:06:56.586059122 (5h40m10.173115343s) [upload=5826]: Got next segment. Inline: true Aug 7 22:06:56.586062599 (5h40m10.173118809s) [upload=5826]: Waiting for error group managing segments... Aug 7 22:06:56.586092645 (5h40m10.173148866s) [upload=5826]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:56.586096352 (5h40m10.173152573s) [upload=5826]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:56.586098696 (5h40m10.173154917s) [upload=5826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:56.650613061 (5h40m10.237669272s) [upload=5826]: Done uploading segments... Aug 7 22:06:56.650627668 (5h40m10.237683889s) [upload=5826]: Upload finished. err: <nil> Aug 7 22:07:58.192526348 (5h41m11.77958259s) [upload=5845]: Starting upload Aug 7 22:07:58.192614623 (5h41m11.779670864s) [upload=5845]: Done starting upload Aug 7 22:07:58.192700764 (5h41m11.779757014s) [upload=5845]: Uploading segments... Aug 7 22:07:58.192744676 (5h41m11.779800917s) [upload=5845]: Got next segment. Inline: true Aug 7 22:07:58.192750096 (5h41m11.779806337s) [upload=5845]: Waiting for error group managing segments... Aug 7 22:07:58.192795772 (5h41m11.779852013s) [upload=5845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:58.192801672 (5h41m11.779857913s) [upload=5845]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:58.192805169 (5h41m11.779861409s) [upload=5845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:58.264015693 (5h41m11.851071934s) [upload=5845]: Done uploading segments... Aug 7 22:07:58.264038786 (5h41m11.851095027s) [upload=5845]: Upload finished. err: <nil> Aug 7 22:09:12.971535319 (5h42m26.558591531s) [upload=5870]: Starting upload Aug 7 22:09:12.971593908 (5h42m26.558650129s) [upload=5870]: Done starting upload Aug 7 22:09:12.971640085 (5h42m26.558696306s) [upload=5870]: Uploading segments... Aug 7 22:09:12.971667336 (5h42m26.558723547s) [upload=5870]: Got next segment. Inline: true Aug 7 22:09:12.971670973 (5h42m26.558727184s) [upload=5870]: Waiting for error group managing segments... Aug 7 22:09:12.971701830 (5h42m26.558758051s) [upload=5870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:12.971705557 (5h42m26.558761768s) [upload=5870]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:12.971707941 (5h42m26.558764152s) [upload=5870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:13.026541326 (5h42m26.613597568s) [upload=5870]: Done uploading segments... Aug 7 22:09:13.026558519 (5h42m26.61361477s) [upload=5870]: Upload finished. err: <nil> Aug 7 22:10:14.601261519 (5h43m28.188317731s) [upload=5887]: Starting upload Aug 7 22:10:14.601309920 (5h43m28.188366141s) [upload=5887]: Done starting upload Aug 7 22:10:14.601350215 (5h43m28.188406436s) [upload=5887]: Uploading segments... Aug 7 22:10:14.601378688 (5h43m28.188434909s) [upload=5887]: Got next segment. Inline: true Aug 7 22:10:14.601382325 (5h43m28.188438546s) [upload=5887]: Waiting for error group managing segments... Aug 7 22:10:14.601411149 (5h43m28.188467361s) [upload=5887]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:14.601414947 (5h43m28.188471168s) [upload=5887]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:14.601417170 (5h43m28.188473391s) [upload=5887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:14.743486813 (5h43m28.330543034s) [upload=5887]: Done uploading segments... Aug 7 22:10:14.743500428 (5h43m28.33055664s) [upload=5887]: Upload finished. err: <nil> Aug 7 22:11:00.429392635 (5h44m14.016448856s) [upload=5899]: Starting upload Aug 7 22:11:00.429436647 (5h44m14.016492868s) [upload=5899]: Done starting upload Aug 7 22:11:00.429474658 (5h44m14.016530879s) [upload=5899]: Uploading segments... Aug 7 22:11:00.429501167 (5h44m14.016557378s) [upload=5899]: Got next segment. Inline: true Aug 7 22:11:00.429504493 (5h44m14.016560714s) [upload=5899]: Waiting for error group managing segments... Aug 7 22:11:00.429533688 (5h44m14.016589909s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:00.429537505 (5h44m14.016593717s) [upload=5899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:00.429540060 (5h44m14.016596281s) [upload=5899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:00.493866886 (5h44m14.080923097s) [upload=5899]: Done uploading segments... Aug 7 22:11:00.493879859 (5h44m14.08093608s) [upload=5899]: Upload finished. err: <nil> Aug 7 22:12:13.235953272 (5h45m26.823009493s) [upload=5915]: Starting upload Aug 7 22:12:13.236008135 (5h45m26.823064356s) [upload=5915]: Done starting upload Aug 7 22:12:13.236053961 (5h45m26.823110172s) [upload=5915]: Uploading segments... Aug 7 22:12:13.236083775 (5h45m26.823139997s) [upload=5915]: Got next segment. Inline: true Aug 7 22:12:13.236087523 (5h45m26.823143733s) [upload=5915]: Waiting for error group managing segments... Aug 7 22:12:13.236120344 (5h45m26.823176555s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:13.236123931 (5h45m26.823180142s) [upload=5915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:13.236126285 (5h45m26.823182506s) [upload=5915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:13.466422283 (5h45m27.053478504s) [upload=5915]: Done uploading segments... Aug 7 22:12:13.466437992 (5h45m27.053494213s) [upload=5915]: Upload finished. err: <nil> Aug 7 22:12:59.930261536 (5h46m13.517317778s) [upload=5926]: Starting upload Aug 7 22:12:59.930344330 (5h46m13.517400582s) [upload=5926]: Done starting upload Aug 7 22:12:59.930411596 (5h46m13.517467837s) [upload=5926]: Uploading segments... Aug 7 22:12:59.930461800 (5h46m13.517518041s) [upload=5926]: Got next segment. Inline: true Aug 7 22:12:59.930467751 (5h46m13.517523992s) [upload=5926]: Waiting for error group managing segments... Aug 7 22:12:59.930514368 (5h46m13.51757061s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:59.930520589 (5h46m13.51757683s) [upload=5926]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:59.930524356 (5h46m13.517580607s) [upload=5926]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:59.913184211 (5h47m13.500240423s) [upload=5926]: Done uploading segments... Aug 7 22:13:59.913207345 (5h47m13.500263566s) [upload=5926]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:00.010650606 (5h47m13.597706828s) [upload=5944]: Starting upload Aug 7 22:14:00.010701831 (5h47m13.597758043s) [upload=5944]: Done starting upload Aug 7 22:14:00.010753608 (5h47m13.59780983s) [upload=5944]: Uploading segments... Aug 7 22:14:00.010809803 (5h47m13.597866024s) [upload=5944]: Got next segment. Inline: true Aug 7 22:14:00.010813610 (5h47m13.597869821s) [upload=5944]: Waiting for error group managing segments... Aug 7 22:14:00.010845560 (5h47m13.597901781s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:00.010849677 (5h47m13.597905899s) [upload=5944]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:00.010851892 (5h47m13.597908113s) [upload=5944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:59.990902753 (5h48m13.577958974s) [upload=5944]: Done uploading segments... Aug 7 22:14:59.990918592 (5h48m13.577974803s) [upload=5944]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.000514999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXX.............................XXXXXX] duration: 30.000687705s FAIL (2025-08-07 22:14:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .737931945s) [upload=5847]: Done uploading segments... Aug 7 22:08:08.150894760 (5h41m21.737951s) [upload=5847]: Upload finished. err: <nil> Aug 7 22:08:48.116259598 (5h42m1.703315819s) [upload=5859]: Starting upload Aug 7 22:08:48.116319110 (5h42m1.703375321s) [upload=5859]: Done starting upload Aug 7 22:08:48.116367930 (5h42m1.703424141s) [upload=5859]: Uploading segments... Aug 7 22:08:48.116414508 (5h42m1.703470779s) [upload=5859]: Got next segment. Inline: true Aug 7 22:08:48.116419497 (5h42m1.703475708s) [upload=5859]: Waiting for error group managing segments... Aug 7 22:08:48.116456857 (5h42m1.703513068s) [upload=5859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:48.116461606 (5h42m1.703517817s) [upload=5859]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:48.116464842 (5h42m1.703521053s) [upload=5859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:48.176815174 (5h42m1.763871415s) [upload=5859]: Done uploading segments... Aug 7 22:08:48.176876088 (5h42m1.763932329s) [upload=5859]: Upload finished. err: <nil> Aug 7 22:09:04.330665274 (5h42m17.917721525s) [upload=5866]: Starting upload Aug 7 22:09:04.330750753 (5h42m17.917806994s) [upload=5866]: Done starting upload Aug 7 22:09:04.330821295 (5h42m17.917877535s) [upload=5866]: Uploading segments... Aug 7 22:09:04.330908929 (5h42m17.91796521s) [upload=5866]: Got next segment. Inline: true Aug 7 22:09:04.330923075 (5h42m17.917979317s) [upload=5866]: Waiting for error group managing segments... Aug 7 22:09:04.330991012 (5h42m17.918047253s) [upload=5866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:04.330999468 (5h42m17.918055718s) [upload=5866]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:04.331005178 (5h42m17.91806143s) [upload=5866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:04.636340599 (5h42m18.22339684s) [upload=5866]: Done uploading segments... Aug 7 22:09:04.636356889 (5h42m18.223413121s) [upload=5866]: Upload finished. err: <nil> Aug 7 22:09:43.052830170 (5h42m56.639886381s) [upload=5877]: Starting upload Aug 7 22:09:43.052883700 (5h42m56.639939921s) [upload=5877]: Done starting upload Aug 7 22:09:43.052934405 (5h42m56.639990626s) [upload=5877]: Uploading segments... Aug 7 22:09:43.052967446 (5h42m56.640023657s) [upload=5877]: Got next segment. Inline: true Aug 7 22:09:43.052971313 (5h42m56.640027534s) [upload=5877]: Waiting for error group managing segments... Aug 7 22:09:43.053000388 (5h42m56.640056609s) [upload=5877]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:43.053004064 (5h42m56.640060276s) [upload=5877]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:43.053006990 (5h42m56.640063211s) [upload=5877]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:43.127325938 (5h42m56.714382178s) [upload=5877]: Done uploading segments... Aug 7 22:09:43.127344483 (5h42m56.714400723s) [upload=5877]: Upload finished. err: <nil> Aug 7 22:10:15.358455928 (5h43m28.945512149s) [upload=5888]: Starting upload Aug 7 22:10:15.358522273 (5h43m28.945578494s) [upload=5888]: Done starting upload Aug 7 22:10:15.358588375 (5h43m28.945644586s) [upload=5888]: Uploading segments... Aug 7 22:10:15.358619654 (5h43m28.945675866s) [upload=5888]: Got next segment. Inline: true Aug 7 22:10:15.358623641 (5h43m28.945679853s) [upload=5888]: Waiting for error group managing segments... Aug 7 22:10:15.358657604 (5h43m28.945713826s) [upload=5888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:15.358661782 (5h43m28.945717993s) [upload=5888]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:15.358665169 (5h43m28.94572139s) [upload=5888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:15.541784918 (5h43m29.128841169s) [upload=5888]: Done uploading segments... Aug 7 22:10:15.541804064 (5h43m29.128860304s) [upload=5888]: Upload finished. err: <nil> Aug 7 22:11:10.710381989 (5h44m24.29743823s) [upload=5902]: Starting upload Aug 7 22:11:10.710469673 (5h44m24.297525913s) [upload=5902]: Done starting upload Aug 7 22:11:10.710537730 (5h44m24.297593971s) [upload=5902]: Uploading segments... Aug 7 22:11:10.710600456 (5h44m24.297656698s) [upload=5902]: Got next segment. Inline: true Aug 7 22:11:10.710607830 (5h44m24.297664072s) [upload=5902]: Waiting for error group managing segments... Aug 7 22:11:10.710658876 (5h44m24.297715118s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:10.710666100 (5h44m24.297722341s) [upload=5902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:10.710670057 (5h44m24.297726299s) [upload=5902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:40.695011486 (5h44m54.282067737s) [upload=5902]: Done uploading segments... Aug 7 22:11:40.695031373 (5h44m54.282087624s) [upload=5902]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:11:40.801935294 (5h44m54.388991515s) [upload=5908]: Starting upload Aug 7 22:11:40.802009452 (5h44m54.389065673s) [upload=5908]: Done starting upload Aug 7 22:11:40.802061539 (5h44m54.38911776s) [upload=5908]: Uploading segments... Aug 7 22:11:40.802108137 (5h44m54.389164358s) [upload=5908]: Got next segment. Inline: true Aug 7 22:11:40.802112685 (5h44m54.389168896s) [upload=5908]: Waiting for error group managing segments... Aug 7 22:11:40.802146128 (5h44m54.389202338s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.802150235 (5h44m54.389206456s) [upload=5908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:40.802172947 (5h44m54.389229168s) [upload=5908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.785746890 (5h45m24.372803111s) [upload=5908]: Done uploading segments... Aug 7 22:12:10.785765736 (5h45m24.372821946s) [upload=5908]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:34.227606352 (5h45m47.814662563s) [upload=5920]: Starting upload Aug 7 22:12:34.227679809 (5h45m47.81473602s) [upload=5920]: Done starting upload Aug 7 22:12:34.227726427 (5h45m47.814782637s) [upload=5920]: Uploading segments... Aug 7 22:12:34.227755791 (5h45m47.814812012s) [upload=5920]: Got next segment. Inline: true Aug 7 22:12:34.227759368 (5h45m47.814815579s) [upload=5920]: Waiting for error group managing segments... Aug 7 22:12:34.227789123 (5h45m47.814845344s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:34.227792860 (5h45m47.814849081s) [upload=5920]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:34.227795325 (5h45m47.814851546s) [upload=5920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:04.214429951 (5h46m17.801486172s) [upload=5920]: Done uploading segments... Aug 7 22:13:04.214444368 (5h46m17.801500579s) [upload=5920]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:21.693003516 (5h46m35.280059727s) [upload=5934]: Starting upload Aug 7 22:13:21.693054191 (5h46m35.280110402s) [upload=5934]: Done starting upload Aug 7 22:13:21.693097442 (5h46m35.280153663s) [upload=5934]: Uploading segments... Aug 7 22:13:21.693124161 (5h46m35.280180382s) [upload=5934]: Got next segment. Inline: true Aug 7 22:13:21.693127627 (5h46m35.280183848s) [upload=5934]: Waiting for error group managing segments... Aug 7 22:13:21.693172341 (5h46m35.280228552s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:21.693176398 (5h46m35.28023262s) [upload=5934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:21.693178873 (5h46m35.280235085s) [upload=5934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:51.677769243 (5h47m5.264825484s) [upload=5934]: Done uploading segments... Aug 7 22:13:51.677786175 (5h47m5.264842416s) [upload=5934]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:51.763566605 (5h47m5.350622826s) [upload=5942]: Starting upload Aug 7 22:13:51.763615116 (5h47m5.350671337s) [upload=5942]: Done starting upload Aug 7 22:13:51.763654910 (5h47m5.350711131s) [upload=5942]: Uploading segments... Aug 7 22:13:51.763683413 (5h47m5.350739634s) [upload=5942]: Got next segment. Inline: true Aug 7 22:13:51.763687000 (5h47m5.350743211s) [upload=5942]: Waiting for error group managing segments... Aug 7 22:13:51.763718620 (5h47m5.350774841s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.763722296 (5h47m5.350778517s) [upload=5942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:51.763724580 (5h47m5.350780801s) [upload=5942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:21.752690510 (5h47m35.339746731s) [upload=5942]: Done uploading segments... Aug 7 22:14:21.752712410 (5h47m35.339768631s) [upload=5942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:21.832761353 (5h47m35.419817564s) [upload=5952]: Starting upload Aug 7 22:14:21.832808701 (5h47m35.419864913s) [upload=5952]: Done starting upload Aug 7 22:14:21.832873814 (5h47m35.419930035s) [upload=5952]: Uploading segments... Aug 7 22:14:21.832905782 (5h47m35.419961993s) [upload=5952]: Got next segment. Inline: true Aug 7 22:14:21.832909440 (5h47m35.419965661s) [upload=5952]: Waiting for error group managing segments... Aug 7 22:14:21.832940607 (5h47m35.419996828s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:21.832944185 (5h47m35.420000406s) [upload=5952]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:21.832946779 (5h47m35.420003s) [upload=5952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:51.821960613 (5h48m5.409016855s) [upload=5952]: Done uploading segments... Aug 7 22:14:51.821995769 (5h48m5.40905202s) [upload=5952]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000704075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.X..................................XXXXXX] duration: 30.000895101s FAIL (2025-08-07 22:14:50 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000906582s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX....................................XXX] duration: 1m0.000662479s FAIL (2025-08-07 22:14:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000676445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX..X........................................XXXX] duration: 30.000751769s FAIL (2025-08-07 22:14:39 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 4807 , piece_id=YPUYDHFEHRTNW4SPSUB7C2CKT677ZCEPQXLD5HITJFY4GWHV4CXQ, seg_pos={0 0}, upload=5906]: Uploading piece... Aug 7 22:11:22.203203428 (5h44m35.790259669s) [noise=true , address=77.90.31.19:4843 , piece_id=VMPEXUKYLQTZFJIKYLSW34FCJE5ZF3NNCJKOIPAJY4U6MZXBLIFQ, seg_pos={0 0}, upload=5906]: Uploading piece... Aug 7 22:11:22.203305939 (5h44m35.79036218s) [noise=true , address=77.90.31.19:4841 , piece_id=WFAYRIGJ3MIGQQS6JLHAWUN7W5KVZYQDHPOBEP5FTV76QONA3XSA, seg_pos={0 0}, upload=5906]: Uploading piece... Aug 7 22:11:22.203450768 (5h44m35.79050698s) [noise=true , address=45.91.116.13:4153 , piece_id=2ITY4VE4TB332VVR56HQ4OJXZ7N7DZ2U442L46PNR24EEEEEDWLA, seg_pos={0 0}, upload=5906]: Uploading piece... Aug 7 22:11:22.203673624 (5h44m35.790729845s) [noise=true , address=45.91.116.11:4118 , piece_id=ETJA6Q4ZE3KHTBEPJEE5FFZOV4M2XW4QVCI5SRN37U2LUYK3TFPA, seg_pos={0 0}, upload=5906]: Uploading piece... Aug 7 22:11:22.205602926 (5h44m35.792659167s) [noise=true , address=104.36.82.14:4222 , piece_id=TQKQYRAYMVIYB7B5LZPJPFGZHRFEBCFFQPBJNOQUXH2WGKGBNETQ, seg_pos={0 0}, upload=5906]: Uploading piece... Aug 7 22:11:22.205825882 (5h44m35.792882093s) [noise=true , address=45.91.116.4:4937 , piece_id=G76MS6JJQSTAP26CACUDXIL6PT722PSLTGFSXU5TWEIGDIUY7MOA, seg_pos={0 0}, upload=5906]: Uploading piece... Aug 7 22:11:22.299257025 (5h44m35.886313236s) [upload=5906]: Next returned nil segment Aug 7 22:11:22.299269118 (5h44m35.886325328s) [upload=5906]: Waiting for error group managing segments... Aug 7 22:11:22.307849063 (5h44m35.894905274s) [noise=true , address=45.91.116.7:4011 , piece_id=MNXDQEHG44ZVPQPXPMKCXXLKGOGPIT3Y4ZSJY6Y7426PACYY5M7Q, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.308032816 (5h44m35.895089067s) [noise=true , address=45.91.116.4:4937 , piece_id=G76MS6JJQSTAP26CACUDXIL6PT722PSLTGFSXU5TWEIGDIUY7MOA, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.308271723 (5h44m35.895327933s) [noise=true , address=45.91.116.9:4262 , piece_id=BSGJX34ZROUCC4YUMA74DQDBYXLTLDMAEC2I6GKKSGTTPLMBZPKQ, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.308874407 (5h44m35.895930628s) [noise=true , address=45.91.116.13:4153 , piece_id=2ITY4VE4TB332VVR56HQ4OJXZ7N7DZ2U442L46PNR24EEEEEDWLA, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.309060865 (5h44m35.896117106s) [noise=true , address=77.90.31.19:4843 , piece_id=VMPEXUKYLQTZFJIKYLSW34FCJE5ZF3NNCJKOIPAJY4U6MZXBLIFQ, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.309264284 (5h44m35.896320506s) [noise=true , address=45.91.116.11:4118 , piece_id=ETJA6Q4ZE3KHTBEPJEE5FFZOV4M2XW4QVCI5SRN37U2LUYK3TFPA, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.309643311 (5h44m35.896699532s) [noise=true , address=45.91.116.5:4299 , piece_id=HQYHBVHRPKBUAELATYGXXLU5YWT3IHLL4XZWAZEAGVBPZZZYHNTQ, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.309901855 (5h44m35.896958095s) [noise=true , address=77.90.31.18:4807 , piece_id=YPUYDHFEHRTNW4SPSUB7C2CKT677ZCEPQXLD5HITJFY4GWHV4CXQ, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.309991903 (5h44m35.897048114s) [noise=true , address=77.90.31.19:4841 , piece_id=WFAYRIGJ3MIGQQS6JLHAWUN7W5KVZYQDHPOBEP5FTV76QONA3XSA, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.312188994 (5h44m35.899245205s) [noise=true , address=104.36.82.5:4838 , piece_id=XFV3PKUD44BJJQQPI4AQU545LD6QC5IULOQC5UTHL6E24ZFNA4AQ, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.312643222 (5h44m35.899699473s) [noise=true , address=77.90.31.11:4554 , piece_id=4S2T5KAAHNR6LCM5MW2OI2OSBLFRAW75U2PXXMN6CH4KXCNGY3YA, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.313852539 (5h44m35.90090875s) [noise=true , address=77.90.31.16:4735 , piece_id=CWMHKST55XRQQAYJZR72MHUPISJPOEQQBYG26CNJ6KIIPDUMY7PA, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.318641760 (5h44m35.905698001s) [noise=true , address=104.36.82.11:4114 , piece_id=MIEUUCW4BN6CPDOR6ZGKH763YFQ6BHJYQIY7P6RDUXBV6SH6NDVA, seg_pos={0 0}, upload=5906]: Done uploading piece. err: <nil> Aug 7 22:11:22.318687725 (5h44m35.905743966s) [seg_pos={0 0}, upload=5906]: Segment reached optimal threshold of 13 pieces. Aug 7 22:11:22.318832305 (5h44m35.905888546s) [noise=true , address=104.36.82.14:4222 , piece_id=TQKQYRAYMVIYB7B5LZPJPFGZHRFEBCFFQPBJNOQUXH2WGKGBNETQ, seg_pos={0 0}, upload=5906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Aug 7 22:11:22.319010858 (5h44m35.906067109s) [noise=true , address=77.90.31.17:4772 , piece_id=O7JU7O3O7OKATXR3VINQFXJTP4FITCEXYY3U67WMECA5TV4YDFWA, seg_pos={0 0}, upload=5906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 5.161.89.252:59916->77.90.31.17:4772: use of closed network connection; write tcp 5.161.89.252:59916->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59916->77.90.31.17:4772: use of closed network connection Aug 7 22:11:22.319030775 (5h44m35.906086986s) [noise=true , address=104.36.82.4:4802 , piece_id=LW2D7QUBROTZ6TRKGI5ZUGNJY52XRXY2AWDUK4B4FRMXNEVBJAWQ, seg_pos={0 0}, upload=5906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 5.161.89.252:56006->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:11:22.319061874 (5h44m35.906118085s) [seg_pos={0 0}, upload=5906]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:11:22.319117337 (5h44m35.906173548s) [upload=5906]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:11:22.319121575 (5h44m35.906177796s) [upload=5906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:11:22.360439841 (5h44m35.947496062s) [upload=5906]: Done uploading segments... Aug 7 22:11:22.360461351 (5h44m35.947517563s) [upload=5906]: Upload finished. err: <nil> Aug 7 22:12:36.266946622 (5h45m49.854002843s) [upload=5921]: Starting upload Aug 7 22:12:36.267004701 (5h45m49.854060932s) [upload=5921]: Done starting upload Aug 7 22:12:36.267090872 (5h45m49.854147113s) [upload=5921]: Uploading segments... Aug 7 22:12:36.267260167 (5h45m49.854316379s) [upload=5921]: Got next segment. Inline: false Aug 7 22:12:36.267292538 (5h45m49.854348749s) [upload=5921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:36.267297517 (5h45m49.854353729s) [upload=5921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:06.251572059 (5h46m19.83862827s) [upload=5921]: Next segment err: context deadline exceeded Aug 7 22:13:06.251626902 (5h46m19.838683123s) [upload=5921]: Done uploading segments... Aug 7 22:13:06.251669180 (5h46m19.838725401s) [upload=5921]: Upload finished. err: context deadline exceeded Aug 7 22:13:06.325530913 (5h46m19.912587155s) [upload=5930]: Starting upload Aug 7 22:13:06.325665455 (5h46m19.912721696s) [upload=5930]: Done starting upload Aug 7 22:13:06.325747527 (5h46m19.912803759s) [upload=5930]: Uploading segments... Aug 7 22:13:06.326389916 (5h46m19.913446158s) [upload=5930]: Got next segment. Inline: false Aug 7 22:13:06.326441924 (5h46m19.913498165s) [upload=5930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:06.326448326 (5h46m19.913504567s) [upload=5930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:36.312414617 (5h46m49.899470828s) [upload=5930]: Next segment err: context deadline exceeded Aug 7 22:13:36.312509364 (5h46m49.899565575s) [upload=5930]: Done uploading segments... Aug 7 22:13:36.312543317 (5h46m49.899599529s) [upload=5930]: Upload finished. err: context deadline exceeded Aug 7 22:13:39.364476764 (5h46m52.951533006s) [upload=5938]: Starting upload Aug 7 22:13:39.364590447 (5h46m52.951646688s) [upload=5938]: Done starting upload Aug 7 22:13:39.364664015 (5h46m52.951720256s) [upload=5938]: Uploading segments... Aug 7 22:13:39.365016743 (5h46m52.952072994s) [upload=5938]: Got next segment. Inline: false Aug 7 22:13:39.365085482 (5h46m52.952141732s) [upload=5938]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:39.365092564 (5h46m52.952148806s) [upload=5938]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:09.350412694 (5h47m22.937468915s) [upload=5938]: Next segment err: context deadline exceeded Aug 7 22:14:09.350534561 (5h47m22.937590782s) [upload=5938]: Done uploading segments... Aug 7 22:14:09.350551452 (5h47m22.937607673s) [upload=5938]: Upload finished. err: context deadline exceeded Aug 7 22:14:09.430002885 (5h47m23.017059106s) [upload=5948]: Starting upload Aug 7 22:14:09.430050244 (5h47m23.017106455s) [upload=5948]: Done starting upload Aug 7 22:14:09.430128399 (5h47m23.017184611s) [upload=5948]: Uploading segments... Aug 7 22:14:09.430278701 (5h47m23.017334912s) [upload=5948]: Got next segment. Inline: false Aug 7 22:14:09.430297376 (5h47m23.017353597s) [upload=5948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:09.430302826 (5h47m23.017359037s) [upload=5948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:39.419494306 (5h47m53.006550557s) [upload=5948]: Next segment err: context deadline exceeded Aug 7 22:14:39.419690903 (5h47m53.006747145s) [upload=5948]: Done uploading segments... Aug 7 22:14:39.419719496 (5h47m53.006775737s) [upload=5948]: Upload finished. err: context deadline exceeded Duration: 30.00076798s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX...X.....X..............................XXXX] duration: 30.001113261s FAIL (2025-08-07 22:14:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001131995s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [..X...X...X.................................................] duration: 401.474026ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:14:32 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 401.541159ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 69.808567ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:14:30 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 69.830528ms edge gateway vhost us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_us_qa_orbiter_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)