Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............................................X...XX...XXXXXX] duration: 30.000978455s FAIL (2025-09-11 21:47:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000991619s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X...XX.] duration: 45.966969435s FAIL (2025-09-11 21:47:53 UTC) context deadline exceeded Duration: 45.96698836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................X.....X....X...XX...] duration: 570.850322ms FAIL (2025-09-11 21:47:41 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== Sep 11 21:45:40.463003522 (319h37m15.81578865s) [noise=true , address=45.91.116.13:4155 , piece_id=7NVZE5YHVQUST2NVF5SZEF6EJPWAFLJGXMQNOSJJUXSFVR5WQ5TA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.463551491 (319h37m15.816336618s) [noise=true , address=45.91.116.5:4298 , piece_id=HZ5CIWWJUEVARBNP47JPB7XFKUQV25PECBIP7VBKGBKOX3PF2LRQ, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.463970126 (319h37m15.816755264s) [noise=true , address=77.90.31.17:4770 , piece_id=QEEK4JZO66OYDRXM54FNHV7JEMLLLQ55ZBKPTXJEIWWQJWJGDO5A, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.464402466 (319h37m15.817187605s) [noise=true , address=145.79.227.14:4839 , piece_id=QEMMRRLOF2TKXY7PJFSLLPGD2VWQ4POZGFOTBZAFLWP4GRWHB7MA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.464802326 (319h37m15.817587465s) [noise=true , address=77.90.31.13:4628 , piece_id=FCGS5G5PJIW5MOYWA52AMHBAV2B34RRTPE3X2BSKFQ5IRTREP4ZA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.892558581 (319h37m16.24534372s) [upload=281456]: Next returned nil segment Sep 11 21:45:40.892580002 (319h37m16.245365129s) [upload=281456]: Waiting for error group managing segments... Sep 11 21:45:41.035387083 (319h37m16.388172221s) [noise=true , address=145.79.227.14:4839 , piece_id=QEMMRRLOF2TKXY7PJFSLLPGD2VWQ4POZGFOTBZAFLWP4GRWHB7MA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:45:41.036541769 (319h37m16.389326896s) [noise=true , address=145.79.227.17:4969 , piece_id=S7JY5DVXUSQBTVHVR44O4AOCKIEFXCAINAJX3XG4NC5RROKIUITQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:45:41.196799468 (319h37m16.549584597s) [noise=true , address=145.79.227.13:4803 , piece_id=VPZ3FZ6OFMHC2X4HK5LBHNMVU2AAMFW6U7EY2ZND43ULWOFTGEDQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:46:10.424396656 (319h37m45.777181764s) [noise=true , address=45.91.116.13:4155 , piece_id=7NVZE5YHVQUST2NVF5SZEF6EJPWAFLJGXMQNOSJJUXSFVR5WQ5TA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection; write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection Sep 11 21:46:10.424472478 (319h37m45.777257576s) [noise=true , address=45.91.116.14:4190 , piece_id=SMJHAECOFSEHXHYMQKY7JLZVYNKT6CXWVHXAXRVVLMD27QBNOQBA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection; write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection Sep 11 21:46:10.424489680 (319h37m45.777274788s) [noise=true , address=45.91.116.11:4119 , piece_id=MAC3C2IVL5JZ2A7NREPRNQ4LIHGXPEPKSKGFHMPXKKB25WO5YOOA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection; write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection Sep 11 21:46:10.424563619 (319h37m45.777348727s) [noise=true , address=77.90.31.15:4697 , piece_id=DV7DTDIM2B7B2DTYD4Y55OTODEJDEL5ZKGEJ452MIGPFEWCMZF7Q, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection; write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection Sep 11 21:46:10.424594106 (319h37m45.777379214s) [noise=true , address=77.90.31.13:4628 , piece_id=FCGS5G5PJIW5MOYWA52AMHBAV2B34RRTPE3X2BSKFQ5IRTREP4ZA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection; write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection Sep 11 21:46:10.424617690 (319h37m45.777402788s) [noise=true , address=45.91.116.9:4261 , piece_id=DVT7RNVBMGVG7ZY3FS3IU3VQZGIKMIY2EUF576JNTP3YZBENLRKQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection; write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection Sep 11 21:46:10.424639601 (319h37m45.777424709s) [noise=true , address=45.91.116.5:4298 , piece_id=HZ5CIWWJUEVARBNP47JPB7XFKUQV25PECBIP7VBKGBKOX3PF2LRQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection; write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection Sep 11 21:46:10.424658607 (319h37m45.777443715s) [noise=true , address=45.91.116.7:4009 , piece_id=6ROEZYTFXYP3MHGFWVTN23CW3HBDD6MCLGRGWMZVCRCVSNEWAYUA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection; write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection Sep 11 21:46:10.424679837 (319h37m45.777464945s) [noise=true , address=77.90.31.17:4770 , piece_id=QEEK4JZO66OYDRXM54FNHV7JEMLLLQ55ZBKPTXJEIWWQJWJGDO5A, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection; write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection Sep 11 21:46:10.424719922 (319h37m45.77750503s) [noise=true , address=45.91.116.12:4226 , piece_id=MYIAGHV7UCQK7QJVPWDTK4AIVDCJ4Z2LETUHNDRPV6U5SXT6BJRA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection; write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection Sep 11 21:46:10.424736864 (319h37m45.777521972s) [noise=true , address=77.90.31.13:4627 , piece_id=UB7DQL5YGYTJIBOO7WGBOW2RI5AEYZRJ475SMGTJSNVHK7HGTUUA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection; write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection Sep 11 21:46:10.424749978 (319h37m45.777535086s) [noise=true , address=77.90.31.19:4841 , piece_id=RUGEOBC7PNZO2HBP2ZBOIXIPZE6FOEME7NQOL72C4DPU7FCOSVMA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection; write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection Sep 11 21:46:10.424807897 (319h37m45.777593005s) [noise=true , address=45.91.116.14:4189 , piece_id=TZPP5WXGVSDMPEELWZ7ZYXNE2WV6PGSC6NCRONPCRFMDIBJLF56A, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: context deadline exceeded; context deadline exceeded Sep 11 21:46:10.424822825 (319h37m45.777607933s) [seg_pos={0 0}, upload=281456]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 11 21:46:10.440712797 (319h37m45.793497905s) [upload=281456]: Done uploading segments... Sep 11 21:46:10.440723677 (319h37m45.793508775s) [upload=281456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 570.883204ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................................X...XX...XXXXX] duration: 30.000918113s FAIL (2025-09-11 21:47:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000952117s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X...XX] duration: 1m0.001454903s FAIL (2025-09-11 21:47:07 UTC) context deadline exceeded Duration: 1m0.001469731s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................X.....X....X...XX..] duration: 2.3019481s FAIL (2025-09-11 21:46:52 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== Sep 11 21:45:40.463003522 (319h37m15.81578865s) [noise=true , address=45.91.116.13:4155 , piece_id=7NVZE5YHVQUST2NVF5SZEF6EJPWAFLJGXMQNOSJJUXSFVR5WQ5TA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.463551491 (319h37m15.816336618s) [noise=true , address=45.91.116.5:4298 , piece_id=HZ5CIWWJUEVARBNP47JPB7XFKUQV25PECBIP7VBKGBKOX3PF2LRQ, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.463970126 (319h37m15.816755264s) [noise=true , address=77.90.31.17:4770 , piece_id=QEEK4JZO66OYDRXM54FNHV7JEMLLLQ55ZBKPTXJEIWWQJWJGDO5A, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.464402466 (319h37m15.817187605s) [noise=true , address=145.79.227.14:4839 , piece_id=QEMMRRLOF2TKXY7PJFSLLPGD2VWQ4POZGFOTBZAFLWP4GRWHB7MA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.464802326 (319h37m15.817587465s) [noise=true , address=77.90.31.13:4628 , piece_id=FCGS5G5PJIW5MOYWA52AMHBAV2B34RRTPE3X2BSKFQ5IRTREP4ZA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.892558581 (319h37m16.24534372s) [upload=281456]: Next returned nil segment Sep 11 21:45:40.892580002 (319h37m16.245365129s) [upload=281456]: Waiting for error group managing segments... Sep 11 21:45:41.035387083 (319h37m16.388172221s) [noise=true , address=145.79.227.14:4839 , piece_id=QEMMRRLOF2TKXY7PJFSLLPGD2VWQ4POZGFOTBZAFLWP4GRWHB7MA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:45:41.036541769 (319h37m16.389326896s) [noise=true , address=145.79.227.17:4969 , piece_id=S7JY5DVXUSQBTVHVR44O4AOCKIEFXCAINAJX3XG4NC5RROKIUITQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:45:41.196799468 (319h37m16.549584597s) [noise=true , address=145.79.227.13:4803 , piece_id=VPZ3FZ6OFMHC2X4HK5LBHNMVU2AAMFW6U7EY2ZND43ULWOFTGEDQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:46:10.424396656 (319h37m45.777181764s) [noise=true , address=45.91.116.13:4155 , piece_id=7NVZE5YHVQUST2NVF5SZEF6EJPWAFLJGXMQNOSJJUXSFVR5WQ5TA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection; write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection Sep 11 21:46:10.424472478 (319h37m45.777257576s) [noise=true , address=45.91.116.14:4190 , piece_id=SMJHAECOFSEHXHYMQKY7JLZVYNKT6CXWVHXAXRVVLMD27QBNOQBA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection; write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection Sep 11 21:46:10.424489680 (319h37m45.777274788s) [noise=true , address=45.91.116.11:4119 , piece_id=MAC3C2IVL5JZ2A7NREPRNQ4LIHGXPEPKSKGFHMPXKKB25WO5YOOA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection; write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection Sep 11 21:46:10.424563619 (319h37m45.777348727s) [noise=true , address=77.90.31.15:4697 , piece_id=DV7DTDIM2B7B2DTYD4Y55OTODEJDEL5ZKGEJ452MIGPFEWCMZF7Q, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection; write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection Sep 11 21:46:10.424594106 (319h37m45.777379214s) [noise=true , address=77.90.31.13:4628 , piece_id=FCGS5G5PJIW5MOYWA52AMHBAV2B34RRTPE3X2BSKFQ5IRTREP4ZA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection; write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection Sep 11 21:46:10.424617690 (319h37m45.777402788s) [noise=true , address=45.91.116.9:4261 , piece_id=DVT7RNVBMGVG7ZY3FS3IU3VQZGIKMIY2EUF576JNTP3YZBENLRKQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection; write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection Sep 11 21:46:10.424639601 (319h37m45.777424709s) [noise=true , address=45.91.116.5:4298 , piece_id=HZ5CIWWJUEVARBNP47JPB7XFKUQV25PECBIP7VBKGBKOX3PF2LRQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection; write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection Sep 11 21:46:10.424658607 (319h37m45.777443715s) [noise=true , address=45.91.116.7:4009 , piece_id=6ROEZYTFXYP3MHGFWVTN23CW3HBDD6MCLGRGWMZVCRCVSNEWAYUA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection; write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection Sep 11 21:46:10.424679837 (319h37m45.777464945s) [noise=true , address=77.90.31.17:4770 , piece_id=QEEK4JZO66OYDRXM54FNHV7JEMLLLQ55ZBKPTXJEIWWQJWJGDO5A, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection; write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection Sep 11 21:46:10.424719922 (319h37m45.77750503s) [noise=true , address=45.91.116.12:4226 , piece_id=MYIAGHV7UCQK7QJVPWDTK4AIVDCJ4Z2LETUHNDRPV6U5SXT6BJRA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection; write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection Sep 11 21:46:10.424736864 (319h37m45.777521972s) [noise=true , address=77.90.31.13:4627 , piece_id=UB7DQL5YGYTJIBOO7WGBOW2RI5AEYZRJ475SMGTJSNVHK7HGTUUA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection; write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection Sep 11 21:46:10.424749978 (319h37m45.777535086s) [noise=true , address=77.90.31.19:4841 , piece_id=RUGEOBC7PNZO2HBP2ZBOIXIPZE6FOEME7NQOL72C4DPU7FCOSVMA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection; write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection Sep 11 21:46:10.424807897 (319h37m45.777593005s) [noise=true , address=45.91.116.14:4189 , piece_id=TZPP5WXGVSDMPEELWZ7ZYXNE2WV6PGSC6NCRONPCRFMDIBJLF56A, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: context deadline exceeded; context deadline exceeded Sep 11 21:46:10.424822825 (319h37m45.777607933s) [seg_pos={0 0}, upload=281456]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 11 21:46:10.440712797 (319h37m45.793497905s) [upload=281456]: Done uploading segments... Sep 11 21:46:10.440723677 (319h37m45.793508775s) [upload=281456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 2.301971644s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................X...XX...XXXX] duration: 30.001614289s FAIL (2025-09-11 21:46:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001639998s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X.....X....X...XX.] duration: 1.270607039s FAIL (2025-09-11 21:46:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== Sep 11 21:45:40.463003522 (319h37m15.81578865s) [noise=true , address=45.91.116.13:4155 , piece_id=7NVZE5YHVQUST2NVF5SZEF6EJPWAFLJGXMQNOSJJUXSFVR5WQ5TA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.463551491 (319h37m15.816336618s) [noise=true , address=45.91.116.5:4298 , piece_id=HZ5CIWWJUEVARBNP47JPB7XFKUQV25PECBIP7VBKGBKOX3PF2LRQ, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.463970126 (319h37m15.816755264s) [noise=true , address=77.90.31.17:4770 , piece_id=QEEK4JZO66OYDRXM54FNHV7JEMLLLQ55ZBKPTXJEIWWQJWJGDO5A, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.464402466 (319h37m15.817187605s) [noise=true , address=145.79.227.14:4839 , piece_id=QEMMRRLOF2TKXY7PJFSLLPGD2VWQ4POZGFOTBZAFLWP4GRWHB7MA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.464802326 (319h37m15.817587465s) [noise=true , address=77.90.31.13:4628 , piece_id=FCGS5G5PJIW5MOYWA52AMHBAV2B34RRTPE3X2BSKFQ5IRTREP4ZA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.892558581 (319h37m16.24534372s) [upload=281456]: Next returned nil segment Sep 11 21:45:40.892580002 (319h37m16.245365129s) [upload=281456]: Waiting for error group managing segments... Sep 11 21:45:41.035387083 (319h37m16.388172221s) [noise=true , address=145.79.227.14:4839 , piece_id=QEMMRRLOF2TKXY7PJFSLLPGD2VWQ4POZGFOTBZAFLWP4GRWHB7MA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:45:41.036541769 (319h37m16.389326896s) [noise=true , address=145.79.227.17:4969 , piece_id=S7JY5DVXUSQBTVHVR44O4AOCKIEFXCAINAJX3XG4NC5RROKIUITQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:45:41.196799468 (319h37m16.549584597s) [noise=true , address=145.79.227.13:4803 , piece_id=VPZ3FZ6OFMHC2X4HK5LBHNMVU2AAMFW6U7EY2ZND43ULWOFTGEDQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:46:10.424396656 (319h37m45.777181764s) [noise=true , address=45.91.116.13:4155 , piece_id=7NVZE5YHVQUST2NVF5SZEF6EJPWAFLJGXMQNOSJJUXSFVR5WQ5TA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection; write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection Sep 11 21:46:10.424472478 (319h37m45.777257576s) [noise=true , address=45.91.116.14:4190 , piece_id=SMJHAECOFSEHXHYMQKY7JLZVYNKT6CXWVHXAXRVVLMD27QBNOQBA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection; write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection Sep 11 21:46:10.424489680 (319h37m45.777274788s) [noise=true , address=45.91.116.11:4119 , piece_id=MAC3C2IVL5JZ2A7NREPRNQ4LIHGXPEPKSKGFHMPXKKB25WO5YOOA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection; write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection Sep 11 21:46:10.424563619 (319h37m45.777348727s) [noise=true , address=77.90.31.15:4697 , piece_id=DV7DTDIM2B7B2DTYD4Y55OTODEJDEL5ZKGEJ452MIGPFEWCMZF7Q, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection; write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection Sep 11 21:46:10.424594106 (319h37m45.777379214s) [noise=true , address=77.90.31.13:4628 , piece_id=FCGS5G5PJIW5MOYWA52AMHBAV2B34RRTPE3X2BSKFQ5IRTREP4ZA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection; write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection Sep 11 21:46:10.424617690 (319h37m45.777402788s) [noise=true , address=45.91.116.9:4261 , piece_id=DVT7RNVBMGVG7ZY3FS3IU3VQZGIKMIY2EUF576JNTP3YZBENLRKQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection; write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection Sep 11 21:46:10.424639601 (319h37m45.777424709s) [noise=true , address=45.91.116.5:4298 , piece_id=HZ5CIWWJUEVARBNP47JPB7XFKUQV25PECBIP7VBKGBKOX3PF2LRQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection; write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection Sep 11 21:46:10.424658607 (319h37m45.777443715s) [noise=true , address=45.91.116.7:4009 , piece_id=6ROEZYTFXYP3MHGFWVTN23CW3HBDD6MCLGRGWMZVCRCVSNEWAYUA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection; write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection Sep 11 21:46:10.424679837 (319h37m45.777464945s) [noise=true , address=77.90.31.17:4770 , piece_id=QEEK4JZO66OYDRXM54FNHV7JEMLLLQ55ZBKPTXJEIWWQJWJGDO5A, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection; write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection Sep 11 21:46:10.424719922 (319h37m45.77750503s) [noise=true , address=45.91.116.12:4226 , piece_id=MYIAGHV7UCQK7QJVPWDTK4AIVDCJ4Z2LETUHNDRPV6U5SXT6BJRA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection; write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection Sep 11 21:46:10.424736864 (319h37m45.777521972s) [noise=true , address=77.90.31.13:4627 , piece_id=UB7DQL5YGYTJIBOO7WGBOW2RI5AEYZRJ475SMGTJSNVHK7HGTUUA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection; write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection Sep 11 21:46:10.424749978 (319h37m45.777535086s) [noise=true , address=77.90.31.19:4841 , piece_id=RUGEOBC7PNZO2HBP2ZBOIXIPZE6FOEME7NQOL72C4DPU7FCOSVMA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection; write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection Sep 11 21:46:10.424807897 (319h37m45.777593005s) [noise=true , address=45.91.116.14:4189 , piece_id=TZPP5WXGVSDMPEELWZ7ZYXNE2WV6PGSC6NCRONPCRFMDIBJLF56A, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: context deadline exceeded; context deadline exceeded Sep 11 21:46:10.424822825 (319h37m45.777607933s) [seg_pos={0 0}, upload=281456]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 11 21:46:10.440712797 (319h37m45.793497905s) [upload=281456]: Done uploading segments... Sep 11 21:46:10.440723677 (319h37m45.793508775s) [upload=281456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 1.270620465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................X.....X....X...XX] duration: 30.017865729s FAIL (2025-09-11 21:46:10 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== Sep 11 21:45:40.463003522 (319h37m15.81578865s) [noise=true , address=45.91.116.13:4155 , piece_id=7NVZE5YHVQUST2NVF5SZEF6EJPWAFLJGXMQNOSJJUXSFVR5WQ5TA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.463551491 (319h37m15.816336618s) [noise=true , address=45.91.116.5:4298 , piece_id=HZ5CIWWJUEVARBNP47JPB7XFKUQV25PECBIP7VBKGBKOX3PF2LRQ, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.463970126 (319h37m15.816755264s) [noise=true , address=77.90.31.17:4770 , piece_id=QEEK4JZO66OYDRXM54FNHV7JEMLLLQ55ZBKPTXJEIWWQJWJGDO5A, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.464402466 (319h37m15.817187605s) [noise=true , address=145.79.227.14:4839 , piece_id=QEMMRRLOF2TKXY7PJFSLLPGD2VWQ4POZGFOTBZAFLWP4GRWHB7MA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.464802326 (319h37m15.817587465s) [noise=true , address=77.90.31.13:4628 , piece_id=FCGS5G5PJIW5MOYWA52AMHBAV2B34RRTPE3X2BSKFQ5IRTREP4ZA, seg_pos={0 0}, upload=281456]: Uploading piece... Sep 11 21:45:40.892558581 (319h37m16.24534372s) [upload=281456]: Next returned nil segment Sep 11 21:45:40.892580002 (319h37m16.245365129s) [upload=281456]: Waiting for error group managing segments... Sep 11 21:45:41.035387083 (319h37m16.388172221s) [noise=true , address=145.79.227.14:4839 , piece_id=QEMMRRLOF2TKXY7PJFSLLPGD2VWQ4POZGFOTBZAFLWP4GRWHB7MA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:45:41.036541769 (319h37m16.389326896s) [noise=true , address=145.79.227.17:4969 , piece_id=S7JY5DVXUSQBTVHVR44O4AOCKIEFXCAINAJX3XG4NC5RROKIUITQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:45:41.196799468 (319h37m16.549584597s) [noise=true , address=145.79.227.13:4803 , piece_id=VPZ3FZ6OFMHC2X4HK5LBHNMVU2AAMFW6U7EY2ZND43ULWOFTGEDQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: <nil> Sep 11 21:46:10.424396656 (319h37m45.777181764s) [noise=true , address=45.91.116.13:4155 , piece_id=7NVZE5YHVQUST2NVF5SZEF6EJPWAFLJGXMQNOSJJUXSFVR5WQ5TA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection; write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:8408->45.91.116.13:4155: use of closed network connection Sep 11 21:46:10.424472478 (319h37m45.777257576s) [noise=true , address=45.91.116.14:4190 , piece_id=SMJHAECOFSEHXHYMQKY7JLZVYNKT6CXWVHXAXRVVLMD27QBNOQBA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection; write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59922->45.91.116.14:4190: use of closed network connection Sep 11 21:46:10.424489680 (319h37m45.777274788s) [noise=true , address=45.91.116.11:4119 , piece_id=MAC3C2IVL5JZ2A7NREPRNQ4LIHGXPEPKSKGFHMPXKKB25WO5YOOA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection; write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:42198->45.91.116.11:4119: use of closed network connection Sep 11 21:46:10.424563619 (319h37m45.777348727s) [noise=true , address=77.90.31.15:4697 , piece_id=DV7DTDIM2B7B2DTYD4Y55OTODEJDEL5ZKGEJ452MIGPFEWCMZF7Q, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection; write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27900->77.90.31.15:4697: use of closed network connection Sep 11 21:46:10.424594106 (319h37m45.777379214s) [noise=true , address=77.90.31.13:4628 , piece_id=FCGS5G5PJIW5MOYWA52AMHBAV2B34RRTPE3X2BSKFQ5IRTREP4ZA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection; write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:18068->77.90.31.13:4628: use of closed network connection Sep 11 21:46:10.424617690 (319h37m45.777402788s) [noise=true , address=45.91.116.9:4261 , piece_id=DVT7RNVBMGVG7ZY3FS3IU3VQZGIKMIY2EUF576JNTP3YZBENLRKQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection; write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54690->45.91.116.9:4261: use of closed network connection Sep 11 21:46:10.424639601 (319h37m45.777424709s) [noise=true , address=45.91.116.5:4298 , piece_id=HZ5CIWWJUEVARBNP47JPB7XFKUQV25PECBIP7VBKGBKOX3PF2LRQ, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection; write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56210->45.91.116.5:4298: use of closed network connection Sep 11 21:46:10.424658607 (319h37m45.777443715s) [noise=true , address=45.91.116.7:4009 , piece_id=6ROEZYTFXYP3MHGFWVTN23CW3HBDD6MCLGRGWMZVCRCVSNEWAYUA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection; write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19310->45.91.116.7:4009: use of closed network connection Sep 11 21:46:10.424679837 (319h37m45.777464945s) [noise=true , address=77.90.31.17:4770 , piece_id=QEEK4JZO66OYDRXM54FNHV7JEMLLLQ55ZBKPTXJEIWWQJWJGDO5A, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection; write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:54410->77.90.31.17:4770: use of closed network connection Sep 11 21:46:10.424719922 (319h37m45.77750503s) [noise=true , address=45.91.116.12:4226 , piece_id=MYIAGHV7UCQK7QJVPWDTK4AIVDCJ4Z2LETUHNDRPV6U5SXT6BJRA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection; write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:44568->45.91.116.12:4226: use of closed network connection Sep 11 21:46:10.424736864 (319h37m45.777521972s) [noise=true , address=77.90.31.13:4627 , piece_id=UB7DQL5YGYTJIBOO7WGBOW2RI5AEYZRJ475SMGTJSNVHK7HGTUUA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection; write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63384->77.90.31.13:4627: use of closed network connection Sep 11 21:46:10.424749978 (319h37m45.777535086s) [noise=true , address=77.90.31.19:4841 , piece_id=RUGEOBC7PNZO2HBP2ZBOIXIPZE6FOEME7NQOL72C4DPU7FCOSVMA, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection; write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27830->77.90.31.19:4841: use of closed network connection Sep 11 21:46:10.424807897 (319h37m45.777593005s) [noise=true , address=45.91.116.14:4189 , piece_id=TZPP5WXGVSDMPEELWZ7ZYXNE2WV6PGSC6NCRONPCRFMDIBJLF56A, seg_pos={0 0}, upload=281456]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: context deadline exceeded; context deadline exceeded Sep 11 21:46:10.424822825 (319h37m45.777607933s) [seg_pos={0 0}, upload=281456]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 11 21:46:10.440712797 (319h37m45.793497905s) [upload=281456]: Done uploading segments... Sep 11 21:46:10.440723677 (319h37m45.793508775s) [upload=281456]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.017888733s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 1m0.000735317s FAIL (2025-09-11 21:46:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000753683s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................X...XX...XXX] duration: 30.000598162s FAIL (2025-09-11 21:45:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000614874s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................X...XX...XX] duration: 30.000567614s FAIL (2025-09-11 21:45:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000598722s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X.....X....X...X] duration: 30.019949929s FAIL (2025-09-11 21:45:03 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 19h36m8.747686649s) [noise=true , address=145.79.227.18:4006 , piece_id=EYLKEJZ5RDPNBPGAUK5VCRM2HRDBG5JWM4ADL5O7A4BTQAC34BJQ, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.395079214 (319h36m8.747864312s) [noise=true , address=45.91.116.9:4263 , piece_id=GLGBE74RNJ3F7NZL7EDY473TK6TJSW45JLTKJBFAV6SXLGVDWYQA, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.395162219 (319h36m8.747947327s) [noise=true , address=77.90.31.18:4805 , piece_id=KBK7HVJRRKZETHXLDM252WFPGDVDCON63SEMQYEJH37LI6ZUAG6Q, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.395317451 (319h36m8.748102559s) [noise=true , address=45.91.116.5:4297 , piece_id=SS2GBXFZZC7MOKC5O2OMPUHSVMJ45GSMCDBKUKEYYLYNDAUE4BXA, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.395395197 (319h36m8.748180295s) [noise=true , address=145.79.227.15:4875 , piece_id=22T7WDGS66N4X56A356ZIX7VLTFD2DDFAOQG4YCYIAYEHMK7P2MQ, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.395649994 (319h36m8.748435102s) [noise=true , address=45.91.116.14:4189 , piece_id=MOF3LEV5QAA43GHHOQXOOXEBN767HX4B7G76MILFFXTPDZF7QBPQ, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.395735374 (319h36m8.748520472s) [noise=true , address=77.90.31.16:4734 , piece_id=FE2CGQYLN4OKMRLLO4VSW5YHWOOHMDNUPTYSHK6NC7RMXKHGL34A, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.395874656 (319h36m8.748659754s) [noise=true , address=77.90.31.17:4771 , piece_id=MI7S2L5Z7YUMCCP5NSYYD5MHSAUGG7LTC2LX4CO3UE5SJIVVN2RA, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.396008146 (319h36m8.748793254s) [noise=true , address=77.90.31.12:4590 , piece_id=X7ELFXPCIQFFCB6GEGZVADDNNRPQAI6DZDT3FI7GJF4R66OO3V7A, seg_pos={0 0}, upload=281439]: Uploading piece... Sep 11 21:44:33.675340066 (319h36m9.028125184s) [upload=281439]: Next returned nil segment Sep 11 21:44:33.675355174 (319h36m9.028140302s) [upload=281439]: Waiting for error group managing segments... Sep 11 21:44:33.768043789 (319h36m9.120828897s) [noise=true , address=145.79.227.15:4875 , piece_id=22T7WDGS66N4X56A356ZIX7VLTFD2DDFAOQG4YCYIAYEHMK7P2MQ, seg_pos={0 0}, upload=281439]: Done uploading piece. err: <nil> Sep 11 21:44:33.803421470 (319h36m9.156206578s) [noise=true , address=145.79.227.2:4115 , piece_id=XJAIC7B6OJWMSAGHJZM2CAHLYKBQ63FNXRDWYMN2C7WGIUW2DAUA, seg_pos={0 0}, upload=281439]: Done uploading piece. err: <nil> Sep 11 21:44:33.820812105 (319h36m9.173597213s) [noise=true , address=145.79.227.5:4221 , piece_id=IJDUW6JQMKTM7A3JY3YPIELKPJTAWPMNRUDSSMFPFEQD5UDXLZIA, seg_pos={0 0}, upload=281439]: Done uploading piece. err: <nil> Sep 11 21:44:34.503260300 (319h36m9.856045408s) [noise=true , address=145.79.227.18:4006 , piece_id=EYLKEJZ5RDPNBPGAUK5VCRM2HRDBG5JWM4ADL5O7A4BTQAC34BJQ, seg_pos={0 0}, upload=281439]: Done uploading piece. err: <nil> Sep 11 21:44:34.884927489 (319h36m10.237712586s) [noise=true , address=145.79.227.19:4078 , piece_id=3XWVBGMRX7DCHPFYCV45YNM72VFHLGNZNU7QDGZVDS7YDOZUAMCQ, seg_pos={0 0}, upload=281439]: Done uploading piece. err: <nil> Sep 11 21:45:03.308161704 (319h36m38.660946812s) [noise=true , address=77.90.31.14:4661 , piece_id=GQIN5A3DROK62S5CIKFPUHVR6N5I3VRZD4PLQY4J7ZDBYYDNT7DA, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:45:03.308318298 (319h36m38.661103416s) [noise=true , address=77.90.31.18:4805 , piece_id=KBK7HVJRRKZETHXLDM252WFPGDVDCON63SEMQYEJH37LI6ZUAG6Q, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 5.161.89.252:57652->77.90.31.18:4805: use of closed network connection; write tcp 5.161.89.252:57652->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:57652->77.90.31.18:4805: use of closed network connection Sep 11 21:45:03.308382047 (319h36m38.661167166s) [noise=true , address=45.91.116.14:4189 , piece_id=MOF3LEV5QAA43GHHOQXOOXEBN767HX4B7G76MILFFXTPDZF7QBPQ, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 5.161.89.252:13644->45.91.116.14:4189: use of closed network connection; write tcp 5.161.89.252:13644->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:13644->45.91.116.14:4189: use of closed network connection Sep 11 21:45:03.308439465 (319h36m38.661224583s) [noise=true , address=77.90.31.16:4734 , piece_id=FE2CGQYLN4OKMRLLO4VSW5YHWOOHMDNUPTYSHK6NC7RMXKHGL34A, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 5.161.89.252:27880->77.90.31.16:4734: use of closed network connection; write tcp 5.161.89.252:27880->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:27880->77.90.31.16:4734: use of closed network connection Sep 11 21:45:03.308492775 (319h36m38.661277894s) [noise=true , address=77.90.31.12:4590 , piece_id=X7ELFXPCIQFFCB6GEGZVADDNNRPQAI6DZDT3FI7GJF4R66OO3V7A, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 5.161.89.252:11232->77.90.31.12:4590: use of closed network connection; write tcp 5.161.89.252:11232->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11232->77.90.31.12:4590: use of closed network connection Sep 11 21:45:03.308501862 (319h36m38.66128697s) [noise=true , address=45.91.116.5:4297 , piece_id=SS2GBXFZZC7MOKC5O2OMPUHSVMJ45GSMCDBKUKEYYLYNDAUE4BXA, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 5.161.89.252:63162->45.91.116.5:4297: use of closed network connection; write tcp 5.161.89.252:63162->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:63162->45.91.116.5:4297: use of closed network connection Sep 11 21:45:03.308530957 (319h36m38.661316065s) [noise=true , address=45.91.116.6:4974 , piece_id=VM3KSQS5TSW242CFTXQYPLD3GADR4HOWGK5D3B7NQAR7DSVONKJQ, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 5.161.89.252:50204->45.91.116.6:4974: use of closed network connection; write tcp 5.161.89.252:50204->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:50204->45.91.116.6:4974: use of closed network connection Sep 11 21:45:03.308555183 (319h36m38.661340291s) [noise=true , address=45.91.116.9:4261 , piece_id=5GZMQKQ567EM3L3BCKJ3NSOEORLD27ZCUFTMB2CQRREPWMTVDHEA, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 5.161.89.252:52848->45.91.116.9:4261: use of closed network connection; write tcp 5.161.89.252:52848->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:52848->45.91.116.9:4261: use of closed network connection Sep 11 21:45:03.308591501 (319h36m38.661376609s) [noise=true , address=77.90.31.19:4841 , piece_id=M4VBFYNWNUO5XGBU3OLNN4LBUKAFKOTH3DZHRWTRJS6TS6Z6GVMA, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 5.161.89.252:26070->77.90.31.19:4841: use of closed network connection; write tcp 5.161.89.252:26070->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:26070->77.90.31.19:4841: use of closed network connection Sep 11 21:45:03.308608743 (319h36m38.661393851s) [noise=true , address=45.91.116.9:4263 , piece_id=GLGBE74RNJ3F7NZL7EDY473TK6TJSW45JLTKJBFAV6SXLGVDWYQA, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 5.161.89.252:10980->45.91.116.9:4263: use of closed network connection; write tcp 5.161.89.252:10980->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:10980->45.91.116.9:4263: use of closed network connection Sep 11 21:45:03.308649539 (319h36m38.661434638s) [noise=true , address=77.90.31.17:4771 , piece_id=MI7S2L5Z7YUMCCP5NSYYD5MHSAUGG7LTC2LX4CO3UE5SJIVVN2RA, seg_pos={0 0}, upload=281439]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 5.161.89.252:59980->77.90.31.17:4771: use of closed network connection; write tcp 5.161.89.252:59980->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:59980->77.90.31.17:4771: use of closed network connection Sep 11 21:45:03.308687160 (319h36m38.661472268s) [seg_pos={0 0}, upload=281439]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Sep 11 21:45:03.327068114 (319h36m38.679853222s) [upload=281439]: Done uploading segments... Sep 11 21:45:03.327086138 (319h36m38.679871246s) [upload=281439]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.019967974s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X...XX...X] duration: 30.000645418s FAIL (2025-09-11 21:44:43 UTC) orbiter warning: context deadline exceeded Duration: 30.000661858s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 702.315007ms FAIL (2025-09-11 21:44:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 702.592667ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.....X....X...] duration: 447.094325ms FAIL (2025-09-11 21:43:27 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== sh batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:40:42.739273081 (319h32m18.092058179s) [upload=281389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:40:42.761685625 (319h32m18.114470723s) [seg_pos={0 0}, upload=281389]: Begin upload segment... Sep 11 21:40:42.761730860 (319h32m18.114515958s) [seg_pos={0 0}, upload=281389]: Done begin upload segment. Sep 11 21:40:42.761750937 (319h32m18.114536036s) [noise=true , address=77.90.31.19:4843 , piece_id=YVJDPAKFLJVXVPMSB6S2UKJUHXKMBMOESYBIOU6VIHTRWRNGYUMQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.761790492 (319h32m18.114575619s) [noise=true , address=77.90.31.19:4842 , piece_id=XVEDDVPEUCHK2YZI3HBXE3SC7RNPRXNB3WZR44FCLAMLL3E3YSPQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762146008 (319h32m18.114931116s) [noise=true , address=45.91.116.8:4048 , piece_id=NZGZEE4HLFZL3BNHGYQBRDOZRUJ4LCWLT4K4TDATT52T5QAKZXYQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762422347 (319h32m18.115207455s) [noise=true , address=145.79.227.16:4912 , piece_id=TNCYV776IVUGPBTAWAD4H3ZHWWAQMRC5OMIVGE46T5MAYYULKOGQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762458083 (319h32m18.115243212s) [noise=true , address=145.79.227.13:4803 , piece_id=LAKGSFTYLSZCJPKBOXTA3QR6WFKV5NAVYG53XZZE326S2HDXFXVQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762714335 (319h32m18.115499432s) [noise=true , address=77.90.31.16:4735 , piece_id=ZTZD5QCVLZBGM2FCIIHGVEFYOSGKSPG3XN6G5JP67V4AGALIN6NA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762998407 (319h32m18.115783515s) [noise=true , address=145.79.227.5:4222 , piece_id=2B5HASIEYHC4BLJT5S5NO6S4BHLJM3273ME2IBMJOPDYN2RG4P2A, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.763374793 (319h32m18.116159921s) [noise=true , address=77.90.31.11:4554 , piece_id=DOE2FHKF3S46CO5DWC56456RM42EUU34K7W2UH2IJJ3L7GQG624A, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.764440451 (319h32m18.117225579s) [noise=true , address=45.91.116.14:4189 , piece_id=CYLWPQOUM7RBCGRT3G4H4CQWPSPYBCP52LGM27KCUBA6C7XKQVJA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.764984481 (319h32m18.117769609s) [noise=true , address=77.90.31.11:4555 , piece_id=H5TTNALKGS324L7UTJZ2LQIBRLTY6MYJLWYCDOG3AVAD5G4H7RJQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.765462658 (319h32m18.118247796s) [noise=true , address=77.90.31.21:4877 , piece_id=72PQCNZFH6XVTCL7OEMGAXERZXMENA3NO2LWJLKACJ2N3OUCZE6Q, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.766161929 (319h32m18.118947057s) [noise=true , address=145.79.227.11:4731 , piece_id=AOZ3WLDJF7U4NY66IVYGELNPWCQLWPRNSM4B5NHLLKF7KZ47GIJA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.766578541 (319h32m18.119363669s) [noise=true , address=77.90.31.18:4805 , piece_id=DNUNHMY6RTWES5FVEZS77UCPNIZX23AN7NBP6VJJOD6BBOWFO35Q, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.772522088 (319h32m18.125307216s) [noise=true , address=45.91.116.12:4227 , piece_id=U6W2BXNCBFBSATTVQOYM7OMR5R2KDWD42DR3RIPAB5EUB5YKMPMQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.773717951 (319h32m18.126503049s) [noise=true , address=77.90.31.14:4663 , piece_id=XI4ONS5JCHWFFG7RYLHFQNOEDEJ5KLN36GUPOEQXFBKGDCZWV3NA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.777409895 (319h32m18.130195003s) [noise=true , address=45.91.116.6:4974 , piece_id=42BFSO4TYR57R56VR3SHJQZ2M36DNMJWUVLYQ7EEF3JWMFCNJXVQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:43.328280614 (319h32m18.681065743s) [upload=281389]: Next returned nil segment Sep 11 21:40:43.328301574 (319h32m18.681086712s) [upload=281389]: Waiting for error group managing segments... Sep 11 21:40:43.618005926 (319h32m18.970791034s) [noise=true , address=145.79.227.11:4731 , piece_id=AOZ3WLDJF7U4NY66IVYGELNPWCQLWPRNSM4B5NHLLKF7KZ47GIJA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.144169037 (319h32m19.496954145s) [noise=true , address=145.79.227.16:4912 , piece_id=TNCYV776IVUGPBTAWAD4H3ZHWWAQMRC5OMIVGE46T5MAYYULKOGQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.216564103 (319h32m19.569349211s) [noise=true , address=77.90.31.21:4877 , piece_id=72PQCNZFH6XVTCL7OEMGAXERZXMENA3NO2LWJLKACJ2N3OUCZE6Q, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.273414814 (319h32m19.626199941s) [noise=true , address=145.79.227.5:4222 , piece_id=2B5HASIEYHC4BLJT5S5NO6S4BHLJM3273ME2IBMJOPDYN2RG4P2A, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.292276579 (319h32m19.645061677s) [noise=true , address=77.90.31.18:4805 , piece_id=DNUNHMY6RTWES5FVEZS77UCPNIZX23AN7NBP6VJJOD6BBOWFO35Q, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:47.706553677 (319h32m23.059338804s) [noise=true , address=145.79.227.13:4803 , piece_id=LAKGSFTYLSZCJPKBOXTA3QR6WFKV5NAVYG53XZZE326S2HDXFXVQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:49.496917908 (319h32m24.849703016s) [noise=true , address=45.91.116.12:4227 , piece_id=U6W2BXNCBFBSATTVQOYM7OMR5R2KDWD42DR3RIPAB5EUB5YKMPMQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:50.509367003 (319h32m25.862152111s) [noise=true , address=77.90.31.16:4735 , piece_id=ZTZD5QCVLZBGM2FCIIHGVEFYOSGKSPG3XN6G5JP67V4AGALIN6NA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:57.373304348 (319h32m32.726089456s) [noise=true , address=77.90.31.19:4842 , piece_id=XVEDDVPEUCHK2YZI3HBXE3SC7RNPRXNB3WZR44FCLAMLL3E3YSPQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:41:12.726155271 (319h32m48.0789404s) [noise=true , address=77.90.31.11:4554 , piece_id=DOE2FHKF3S46CO5DWC56456RM42EUU34K7W2UH2IJJ3L7GQG624A, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:41:12.726171722 (319h32m48.07895686s) [noise=true , address=45.91.116.6:4974 , piece_id=42BFSO4TYR57R56VR3SHJQZ2M36DNMJWUVLYQ7EEF3JWMFCNJXVQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:41:12.726468989 (319h32m48.079254118s) [noise=true , address=77.90.31.19:4843 , piece_id=YVJDPAKFLJVXVPMSB6S2UKJUHXKMBMOESYBIOU6VIHTRWRNGYUMQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection Sep 11 21:41:12.726507482 (319h32m48.07929262s) [noise=true , address=77.90.31.14:4663 , piece_id=XI4ONS5JCHWFFG7RYLHFQNOEDEJ5KLN36GUPOEQXFBKGDCZWV3NA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: expected piece hash; write tcp 5.161.89.252:54134->77.90.31.14:4663: use of closed network connection; write tcp 5.161.89.252:54134->77.90.31.14:4663: use of closed network connection; EOF Sep 11 21:41:12.726593113 (319h32m48.07937824s) [noise=true , address=45.91.116.8:4048 , piece_id=NZGZEE4HLFZL3BNHGYQBRDOZRUJ4LCWLT4K4TDATT52T5QAKZXYQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection; write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection Sep 11 21:41:12.726644118 (319h32m48.079429246s) [noise=true , address=77.90.31.11:4555 , piece_id=H5TTNALKGS324L7UTJZ2LQIBRLTY6MYJLWYCDOG3AVAD5G4H7RJQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection; write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection Sep 11 21:41:12.726683552 (319h32m48.07946869s) [noise=true , address=45.91.116.14:4189 , piece_id=CYLWPQOUM7RBCGRT3G4H4CQWPSPYBCP52LGM27KCUBA6C7XKQVJA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection; write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection Sep 11 21:41:12.726703188 (319h32m48.079488327s) [seg_pos={0 0}, upload=281389]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:41:12.743090843 (319h32m48.095875971s) [upload=281389]: Done uploading segments... Sep 11 21:41:12.743111792 (319h32m48.095896921s) [upload=281389]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 447.122017ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X...XX...] duration: 436.2668ms FAIL (2025-09-11 21:43:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 436.286096ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.713719842s FAIL (2025-09-11 21:43:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.713741682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X...XX..] duration: 656.647726ms FAIL (2025-09-11 21:43:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 656.666541ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.....X....X..] duration: 529.159977ms FAIL (2025-09-11 21:42:47 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== sh batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:40:42.739273081 (319h32m18.092058179s) [upload=281389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:40:42.761685625 (319h32m18.114470723s) [seg_pos={0 0}, upload=281389]: Begin upload segment... Sep 11 21:40:42.761730860 (319h32m18.114515958s) [seg_pos={0 0}, upload=281389]: Done begin upload segment. Sep 11 21:40:42.761750937 (319h32m18.114536036s) [noise=true , address=77.90.31.19:4843 , piece_id=YVJDPAKFLJVXVPMSB6S2UKJUHXKMBMOESYBIOU6VIHTRWRNGYUMQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.761790492 (319h32m18.114575619s) [noise=true , address=77.90.31.19:4842 , piece_id=XVEDDVPEUCHK2YZI3HBXE3SC7RNPRXNB3WZR44FCLAMLL3E3YSPQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762146008 (319h32m18.114931116s) [noise=true , address=45.91.116.8:4048 , piece_id=NZGZEE4HLFZL3BNHGYQBRDOZRUJ4LCWLT4K4TDATT52T5QAKZXYQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762422347 (319h32m18.115207455s) [noise=true , address=145.79.227.16:4912 , piece_id=TNCYV776IVUGPBTAWAD4H3ZHWWAQMRC5OMIVGE46T5MAYYULKOGQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762458083 (319h32m18.115243212s) [noise=true , address=145.79.227.13:4803 , piece_id=LAKGSFTYLSZCJPKBOXTA3QR6WFKV5NAVYG53XZZE326S2HDXFXVQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762714335 (319h32m18.115499432s) [noise=true , address=77.90.31.16:4735 , piece_id=ZTZD5QCVLZBGM2FCIIHGVEFYOSGKSPG3XN6G5JP67V4AGALIN6NA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762998407 (319h32m18.115783515s) [noise=true , address=145.79.227.5:4222 , piece_id=2B5HASIEYHC4BLJT5S5NO6S4BHLJM3273ME2IBMJOPDYN2RG4P2A, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.763374793 (319h32m18.116159921s) [noise=true , address=77.90.31.11:4554 , piece_id=DOE2FHKF3S46CO5DWC56456RM42EUU34K7W2UH2IJJ3L7GQG624A, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.764440451 (319h32m18.117225579s) [noise=true , address=45.91.116.14:4189 , piece_id=CYLWPQOUM7RBCGRT3G4H4CQWPSPYBCP52LGM27KCUBA6C7XKQVJA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.764984481 (319h32m18.117769609s) [noise=true , address=77.90.31.11:4555 , piece_id=H5TTNALKGS324L7UTJZ2LQIBRLTY6MYJLWYCDOG3AVAD5G4H7RJQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.765462658 (319h32m18.118247796s) [noise=true , address=77.90.31.21:4877 , piece_id=72PQCNZFH6XVTCL7OEMGAXERZXMENA3NO2LWJLKACJ2N3OUCZE6Q, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.766161929 (319h32m18.118947057s) [noise=true , address=145.79.227.11:4731 , piece_id=AOZ3WLDJF7U4NY66IVYGELNPWCQLWPRNSM4B5NHLLKF7KZ47GIJA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.766578541 (319h32m18.119363669s) [noise=true , address=77.90.31.18:4805 , piece_id=DNUNHMY6RTWES5FVEZS77UCPNIZX23AN7NBP6VJJOD6BBOWFO35Q, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.772522088 (319h32m18.125307216s) [noise=true , address=45.91.116.12:4227 , piece_id=U6W2BXNCBFBSATTVQOYM7OMR5R2KDWD42DR3RIPAB5EUB5YKMPMQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.773717951 (319h32m18.126503049s) [noise=true , address=77.90.31.14:4663 , piece_id=XI4ONS5JCHWFFG7RYLHFQNOEDEJ5KLN36GUPOEQXFBKGDCZWV3NA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.777409895 (319h32m18.130195003s) [noise=true , address=45.91.116.6:4974 , piece_id=42BFSO4TYR57R56VR3SHJQZ2M36DNMJWUVLYQ7EEF3JWMFCNJXVQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:43.328280614 (319h32m18.681065743s) [upload=281389]: Next returned nil segment Sep 11 21:40:43.328301574 (319h32m18.681086712s) [upload=281389]: Waiting for error group managing segments... Sep 11 21:40:43.618005926 (319h32m18.970791034s) [noise=true , address=145.79.227.11:4731 , piece_id=AOZ3WLDJF7U4NY66IVYGELNPWCQLWPRNSM4B5NHLLKF7KZ47GIJA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.144169037 (319h32m19.496954145s) [noise=true , address=145.79.227.16:4912 , piece_id=TNCYV776IVUGPBTAWAD4H3ZHWWAQMRC5OMIVGE46T5MAYYULKOGQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.216564103 (319h32m19.569349211s) [noise=true , address=77.90.31.21:4877 , piece_id=72PQCNZFH6XVTCL7OEMGAXERZXMENA3NO2LWJLKACJ2N3OUCZE6Q, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.273414814 (319h32m19.626199941s) [noise=true , address=145.79.227.5:4222 , piece_id=2B5HASIEYHC4BLJT5S5NO6S4BHLJM3273ME2IBMJOPDYN2RG4P2A, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.292276579 (319h32m19.645061677s) [noise=true , address=77.90.31.18:4805 , piece_id=DNUNHMY6RTWES5FVEZS77UCPNIZX23AN7NBP6VJJOD6BBOWFO35Q, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:47.706553677 (319h32m23.059338804s) [noise=true , address=145.79.227.13:4803 , piece_id=LAKGSFTYLSZCJPKBOXTA3QR6WFKV5NAVYG53XZZE326S2HDXFXVQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:49.496917908 (319h32m24.849703016s) [noise=true , address=45.91.116.12:4227 , piece_id=U6W2BXNCBFBSATTVQOYM7OMR5R2KDWD42DR3RIPAB5EUB5YKMPMQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:50.509367003 (319h32m25.862152111s) [noise=true , address=77.90.31.16:4735 , piece_id=ZTZD5QCVLZBGM2FCIIHGVEFYOSGKSPG3XN6G5JP67V4AGALIN6NA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:57.373304348 (319h32m32.726089456s) [noise=true , address=77.90.31.19:4842 , piece_id=XVEDDVPEUCHK2YZI3HBXE3SC7RNPRXNB3WZR44FCLAMLL3E3YSPQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:41:12.726155271 (319h32m48.0789404s) [noise=true , address=77.90.31.11:4554 , piece_id=DOE2FHKF3S46CO5DWC56456RM42EUU34K7W2UH2IJJ3L7GQG624A, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:41:12.726171722 (319h32m48.07895686s) [noise=true , address=45.91.116.6:4974 , piece_id=42BFSO4TYR57R56VR3SHJQZ2M36DNMJWUVLYQ7EEF3JWMFCNJXVQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:41:12.726468989 (319h32m48.079254118s) [noise=true , address=77.90.31.19:4843 , piece_id=YVJDPAKFLJVXVPMSB6S2UKJUHXKMBMOESYBIOU6VIHTRWRNGYUMQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection Sep 11 21:41:12.726507482 (319h32m48.07929262s) [noise=true , address=77.90.31.14:4663 , piece_id=XI4ONS5JCHWFFG7RYLHFQNOEDEJ5KLN36GUPOEQXFBKGDCZWV3NA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: expected piece hash; write tcp 5.161.89.252:54134->77.90.31.14:4663: use of closed network connection; write tcp 5.161.89.252:54134->77.90.31.14:4663: use of closed network connection; EOF Sep 11 21:41:12.726593113 (319h32m48.07937824s) [noise=true , address=45.91.116.8:4048 , piece_id=NZGZEE4HLFZL3BNHGYQBRDOZRUJ4LCWLT4K4TDATT52T5QAKZXYQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection; write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection Sep 11 21:41:12.726644118 (319h32m48.079429246s) [noise=true , address=77.90.31.11:4555 , piece_id=H5TTNALKGS324L7UTJZ2LQIBRLTY6MYJLWYCDOG3AVAD5G4H7RJQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection; write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection Sep 11 21:41:12.726683552 (319h32m48.07946869s) [noise=true , address=45.91.116.14:4189 , piece_id=CYLWPQOUM7RBCGRT3G4H4CQWPSPYBCP52LGM27KCUBA6C7XKQVJA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection; write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection Sep 11 21:41:12.726703188 (319h32m48.079488327s) [seg_pos={0 0}, upload=281389]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:41:12.743090843 (319h32m48.095875971s) [upload=281389]: Done uploading segments... Sep 11 21:41:12.743111792 (319h32m48.095896921s) [upload=281389]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 529.184332ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 17.577042993s FAIL (2025-09-11 21:42:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.577060987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X...XX.] duration: 490.651947ms FAIL (2025-09-11 21:42:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 490.670812ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X...XX] duration: 30.001097619s FAIL (2025-09-11 21:42:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001112487s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000414443s FAIL (2025-09-11 21:42:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000427288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.....X....X.] duration: 1.39509763s FAIL (2025-09-11 21:41:40 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== sh batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:40:42.739273081 (319h32m18.092058179s) [upload=281389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:40:42.761685625 (319h32m18.114470723s) [seg_pos={0 0}, upload=281389]: Begin upload segment... Sep 11 21:40:42.761730860 (319h32m18.114515958s) [seg_pos={0 0}, upload=281389]: Done begin upload segment. Sep 11 21:40:42.761750937 (319h32m18.114536036s) [noise=true , address=77.90.31.19:4843 , piece_id=YVJDPAKFLJVXVPMSB6S2UKJUHXKMBMOESYBIOU6VIHTRWRNGYUMQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.761790492 (319h32m18.114575619s) [noise=true , address=77.90.31.19:4842 , piece_id=XVEDDVPEUCHK2YZI3HBXE3SC7RNPRXNB3WZR44FCLAMLL3E3YSPQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762146008 (319h32m18.114931116s) [noise=true , address=45.91.116.8:4048 , piece_id=NZGZEE4HLFZL3BNHGYQBRDOZRUJ4LCWLT4K4TDATT52T5QAKZXYQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762422347 (319h32m18.115207455s) [noise=true , address=145.79.227.16:4912 , piece_id=TNCYV776IVUGPBTAWAD4H3ZHWWAQMRC5OMIVGE46T5MAYYULKOGQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762458083 (319h32m18.115243212s) [noise=true , address=145.79.227.13:4803 , piece_id=LAKGSFTYLSZCJPKBOXTA3QR6WFKV5NAVYG53XZZE326S2HDXFXVQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762714335 (319h32m18.115499432s) [noise=true , address=77.90.31.16:4735 , piece_id=ZTZD5QCVLZBGM2FCIIHGVEFYOSGKSPG3XN6G5JP67V4AGALIN6NA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762998407 (319h32m18.115783515s) [noise=true , address=145.79.227.5:4222 , piece_id=2B5HASIEYHC4BLJT5S5NO6S4BHLJM3273ME2IBMJOPDYN2RG4P2A, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.763374793 (319h32m18.116159921s) [noise=true , address=77.90.31.11:4554 , piece_id=DOE2FHKF3S46CO5DWC56456RM42EUU34K7W2UH2IJJ3L7GQG624A, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.764440451 (319h32m18.117225579s) [noise=true , address=45.91.116.14:4189 , piece_id=CYLWPQOUM7RBCGRT3G4H4CQWPSPYBCP52LGM27KCUBA6C7XKQVJA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.764984481 (319h32m18.117769609s) [noise=true , address=77.90.31.11:4555 , piece_id=H5TTNALKGS324L7UTJZ2LQIBRLTY6MYJLWYCDOG3AVAD5G4H7RJQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.765462658 (319h32m18.118247796s) [noise=true , address=77.90.31.21:4877 , piece_id=72PQCNZFH6XVTCL7OEMGAXERZXMENA3NO2LWJLKACJ2N3OUCZE6Q, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.766161929 (319h32m18.118947057s) [noise=true , address=145.79.227.11:4731 , piece_id=AOZ3WLDJF7U4NY66IVYGELNPWCQLWPRNSM4B5NHLLKF7KZ47GIJA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.766578541 (319h32m18.119363669s) [noise=true , address=77.90.31.18:4805 , piece_id=DNUNHMY6RTWES5FVEZS77UCPNIZX23AN7NBP6VJJOD6BBOWFO35Q, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.772522088 (319h32m18.125307216s) [noise=true , address=45.91.116.12:4227 , piece_id=U6W2BXNCBFBSATTVQOYM7OMR5R2KDWD42DR3RIPAB5EUB5YKMPMQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.773717951 (319h32m18.126503049s) [noise=true , address=77.90.31.14:4663 , piece_id=XI4ONS5JCHWFFG7RYLHFQNOEDEJ5KLN36GUPOEQXFBKGDCZWV3NA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.777409895 (319h32m18.130195003s) [noise=true , address=45.91.116.6:4974 , piece_id=42BFSO4TYR57R56VR3SHJQZ2M36DNMJWUVLYQ7EEF3JWMFCNJXVQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:43.328280614 (319h32m18.681065743s) [upload=281389]: Next returned nil segment Sep 11 21:40:43.328301574 (319h32m18.681086712s) [upload=281389]: Waiting for error group managing segments... Sep 11 21:40:43.618005926 (319h32m18.970791034s) [noise=true , address=145.79.227.11:4731 , piece_id=AOZ3WLDJF7U4NY66IVYGELNPWCQLWPRNSM4B5NHLLKF7KZ47GIJA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.144169037 (319h32m19.496954145s) [noise=true , address=145.79.227.16:4912 , piece_id=TNCYV776IVUGPBTAWAD4H3ZHWWAQMRC5OMIVGE46T5MAYYULKOGQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.216564103 (319h32m19.569349211s) [noise=true , address=77.90.31.21:4877 , piece_id=72PQCNZFH6XVTCL7OEMGAXERZXMENA3NO2LWJLKACJ2N3OUCZE6Q, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.273414814 (319h32m19.626199941s) [noise=true , address=145.79.227.5:4222 , piece_id=2B5HASIEYHC4BLJT5S5NO6S4BHLJM3273ME2IBMJOPDYN2RG4P2A, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.292276579 (319h32m19.645061677s) [noise=true , address=77.90.31.18:4805 , piece_id=DNUNHMY6RTWES5FVEZS77UCPNIZX23AN7NBP6VJJOD6BBOWFO35Q, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:47.706553677 (319h32m23.059338804s) [noise=true , address=145.79.227.13:4803 , piece_id=LAKGSFTYLSZCJPKBOXTA3QR6WFKV5NAVYG53XZZE326S2HDXFXVQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:49.496917908 (319h32m24.849703016s) [noise=true , address=45.91.116.12:4227 , piece_id=U6W2BXNCBFBSATTVQOYM7OMR5R2KDWD42DR3RIPAB5EUB5YKMPMQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:50.509367003 (319h32m25.862152111s) [noise=true , address=77.90.31.16:4735 , piece_id=ZTZD5QCVLZBGM2FCIIHGVEFYOSGKSPG3XN6G5JP67V4AGALIN6NA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:57.373304348 (319h32m32.726089456s) [noise=true , address=77.90.31.19:4842 , piece_id=XVEDDVPEUCHK2YZI3HBXE3SC7RNPRXNB3WZR44FCLAMLL3E3YSPQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:41:12.726155271 (319h32m48.0789404s) [noise=true , address=77.90.31.11:4554 , piece_id=DOE2FHKF3S46CO5DWC56456RM42EUU34K7W2UH2IJJ3L7GQG624A, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:41:12.726171722 (319h32m48.07895686s) [noise=true , address=45.91.116.6:4974 , piece_id=42BFSO4TYR57R56VR3SHJQZ2M36DNMJWUVLYQ7EEF3JWMFCNJXVQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:41:12.726468989 (319h32m48.079254118s) [noise=true , address=77.90.31.19:4843 , piece_id=YVJDPAKFLJVXVPMSB6S2UKJUHXKMBMOESYBIOU6VIHTRWRNGYUMQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection Sep 11 21:41:12.726507482 (319h32m48.07929262s) [noise=true , address=77.90.31.14:4663 , piece_id=XI4ONS5JCHWFFG7RYLHFQNOEDEJ5KLN36GUPOEQXFBKGDCZWV3NA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: expected piece hash; write tcp 5.161.89.252:54134->77.90.31.14:4663: use of closed network connection; write tcp 5.161.89.252:54134->77.90.31.14:4663: use of closed network connection; EOF Sep 11 21:41:12.726593113 (319h32m48.07937824s) [noise=true , address=45.91.116.8:4048 , piece_id=NZGZEE4HLFZL3BNHGYQBRDOZRUJ4LCWLT4K4TDATT52T5QAKZXYQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection; write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection Sep 11 21:41:12.726644118 (319h32m48.079429246s) [noise=true , address=77.90.31.11:4555 , piece_id=H5TTNALKGS324L7UTJZ2LQIBRLTY6MYJLWYCDOG3AVAD5G4H7RJQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection; write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection Sep 11 21:41:12.726683552 (319h32m48.07946869s) [noise=true , address=45.91.116.14:4189 , piece_id=CYLWPQOUM7RBCGRT3G4H4CQWPSPYBCP52LGM27KCUBA6C7XKQVJA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection; write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection Sep 11 21:41:12.726703188 (319h32m48.079488327s) [seg_pos={0 0}, upload=281389]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:41:12.743090843 (319h32m48.095875971s) [upload=281389]: Done uploading segments... Sep 11 21:41:12.743111792 (319h32m48.095896921s) [upload=281389]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 1.395117628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.....X....X] duration: 30.019176707s FAIL (2025-09-11 21:41:12 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== sh batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:40:42.739273081 (319h32m18.092058179s) [upload=281389]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:40:42.761685625 (319h32m18.114470723s) [seg_pos={0 0}, upload=281389]: Begin upload segment... Sep 11 21:40:42.761730860 (319h32m18.114515958s) [seg_pos={0 0}, upload=281389]: Done begin upload segment. Sep 11 21:40:42.761750937 (319h32m18.114536036s) [noise=true , address=77.90.31.19:4843 , piece_id=YVJDPAKFLJVXVPMSB6S2UKJUHXKMBMOESYBIOU6VIHTRWRNGYUMQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.761790492 (319h32m18.114575619s) [noise=true , address=77.90.31.19:4842 , piece_id=XVEDDVPEUCHK2YZI3HBXE3SC7RNPRXNB3WZR44FCLAMLL3E3YSPQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762146008 (319h32m18.114931116s) [noise=true , address=45.91.116.8:4048 , piece_id=NZGZEE4HLFZL3BNHGYQBRDOZRUJ4LCWLT4K4TDATT52T5QAKZXYQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762422347 (319h32m18.115207455s) [noise=true , address=145.79.227.16:4912 , piece_id=TNCYV776IVUGPBTAWAD4H3ZHWWAQMRC5OMIVGE46T5MAYYULKOGQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762458083 (319h32m18.115243212s) [noise=true , address=145.79.227.13:4803 , piece_id=LAKGSFTYLSZCJPKBOXTA3QR6WFKV5NAVYG53XZZE326S2HDXFXVQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762714335 (319h32m18.115499432s) [noise=true , address=77.90.31.16:4735 , piece_id=ZTZD5QCVLZBGM2FCIIHGVEFYOSGKSPG3XN6G5JP67V4AGALIN6NA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.762998407 (319h32m18.115783515s) [noise=true , address=145.79.227.5:4222 , piece_id=2B5HASIEYHC4BLJT5S5NO6S4BHLJM3273ME2IBMJOPDYN2RG4P2A, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.763374793 (319h32m18.116159921s) [noise=true , address=77.90.31.11:4554 , piece_id=DOE2FHKF3S46CO5DWC56456RM42EUU34K7W2UH2IJJ3L7GQG624A, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.764440451 (319h32m18.117225579s) [noise=true , address=45.91.116.14:4189 , piece_id=CYLWPQOUM7RBCGRT3G4H4CQWPSPYBCP52LGM27KCUBA6C7XKQVJA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.764984481 (319h32m18.117769609s) [noise=true , address=77.90.31.11:4555 , piece_id=H5TTNALKGS324L7UTJZ2LQIBRLTY6MYJLWYCDOG3AVAD5G4H7RJQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.765462658 (319h32m18.118247796s) [noise=true , address=77.90.31.21:4877 , piece_id=72PQCNZFH6XVTCL7OEMGAXERZXMENA3NO2LWJLKACJ2N3OUCZE6Q, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.766161929 (319h32m18.118947057s) [noise=true , address=145.79.227.11:4731 , piece_id=AOZ3WLDJF7U4NY66IVYGELNPWCQLWPRNSM4B5NHLLKF7KZ47GIJA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.766578541 (319h32m18.119363669s) [noise=true , address=77.90.31.18:4805 , piece_id=DNUNHMY6RTWES5FVEZS77UCPNIZX23AN7NBP6VJJOD6BBOWFO35Q, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.772522088 (319h32m18.125307216s) [noise=true , address=45.91.116.12:4227 , piece_id=U6W2BXNCBFBSATTVQOYM7OMR5R2KDWD42DR3RIPAB5EUB5YKMPMQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.773717951 (319h32m18.126503049s) [noise=true , address=77.90.31.14:4663 , piece_id=XI4ONS5JCHWFFG7RYLHFQNOEDEJ5KLN36GUPOEQXFBKGDCZWV3NA, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:42.777409895 (319h32m18.130195003s) [noise=true , address=45.91.116.6:4974 , piece_id=42BFSO4TYR57R56VR3SHJQZ2M36DNMJWUVLYQ7EEF3JWMFCNJXVQ, seg_pos={0 0}, upload=281389]: Uploading piece... Sep 11 21:40:43.328280614 (319h32m18.681065743s) [upload=281389]: Next returned nil segment Sep 11 21:40:43.328301574 (319h32m18.681086712s) [upload=281389]: Waiting for error group managing segments... Sep 11 21:40:43.618005926 (319h32m18.970791034s) [noise=true , address=145.79.227.11:4731 , piece_id=AOZ3WLDJF7U4NY66IVYGELNPWCQLWPRNSM4B5NHLLKF7KZ47GIJA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.144169037 (319h32m19.496954145s) [noise=true , address=145.79.227.16:4912 , piece_id=TNCYV776IVUGPBTAWAD4H3ZHWWAQMRC5OMIVGE46T5MAYYULKOGQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.216564103 (319h32m19.569349211s) [noise=true , address=77.90.31.21:4877 , piece_id=72PQCNZFH6XVTCL7OEMGAXERZXMENA3NO2LWJLKACJ2N3OUCZE6Q, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.273414814 (319h32m19.626199941s) [noise=true , address=145.79.227.5:4222 , piece_id=2B5HASIEYHC4BLJT5S5NO6S4BHLJM3273ME2IBMJOPDYN2RG4P2A, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:44.292276579 (319h32m19.645061677s) [noise=true , address=77.90.31.18:4805 , piece_id=DNUNHMY6RTWES5FVEZS77UCPNIZX23AN7NBP6VJJOD6BBOWFO35Q, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:47.706553677 (319h32m23.059338804s) [noise=true , address=145.79.227.13:4803 , piece_id=LAKGSFTYLSZCJPKBOXTA3QR6WFKV5NAVYG53XZZE326S2HDXFXVQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:49.496917908 (319h32m24.849703016s) [noise=true , address=45.91.116.12:4227 , piece_id=U6W2BXNCBFBSATTVQOYM7OMR5R2KDWD42DR3RIPAB5EUB5YKMPMQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:50.509367003 (319h32m25.862152111s) [noise=true , address=77.90.31.16:4735 , piece_id=ZTZD5QCVLZBGM2FCIIHGVEFYOSGKSPG3XN6G5JP67V4AGALIN6NA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:40:57.373304348 (319h32m32.726089456s) [noise=true , address=77.90.31.19:4842 , piece_id=XVEDDVPEUCHK2YZI3HBXE3SC7RNPRXNB3WZR44FCLAMLL3E3YSPQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: <nil> Sep 11 21:41:12.726155271 (319h32m48.0789404s) [noise=true , address=77.90.31.11:4554 , piece_id=DOE2FHKF3S46CO5DWC56456RM42EUU34K7W2UH2IJJ3L7GQG624A, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:41:12.726171722 (319h32m48.07895686s) [noise=true , address=45.91.116.6:4974 , piece_id=42BFSO4TYR57R56VR3SHJQZ2M36DNMJWUVLYQ7EEF3JWMFCNJXVQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: expected piece hash; context deadline exceeded; EOF Sep 11 21:41:12.726468989 (319h32m48.079254118s) [noise=true , address=77.90.31.19:4843 , piece_id=YVJDPAKFLJVXVPMSB6S2UKJUHXKMBMOESYBIOU6VIHTRWRNGYUMQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:61726->77.90.31.19:4843: use of closed network connection Sep 11 21:41:12.726507482 (319h32m48.07929262s) [noise=true , address=77.90.31.14:4663 , piece_id=XI4ONS5JCHWFFG7RYLHFQNOEDEJ5KLN36GUPOEQXFBKGDCZWV3NA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: expected piece hash; write tcp 5.161.89.252:54134->77.90.31.14:4663: use of closed network connection; write tcp 5.161.89.252:54134->77.90.31.14:4663: use of closed network connection; EOF Sep 11 21:41:12.726593113 (319h32m48.07937824s) [noise=true , address=45.91.116.8:4048 , piece_id=NZGZEE4HLFZL3BNHGYQBRDOZRUJ4LCWLT4K4TDATT52T5QAKZXYQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection; write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:44294->45.91.116.8:4048: use of closed network connection Sep 11 21:41:12.726644118 (319h32m48.079429246s) [noise=true , address=77.90.31.11:4555 , piece_id=H5TTNALKGS324L7UTJZ2LQIBRLTY6MYJLWYCDOG3AVAD5G4H7RJQ, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection; write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49250->77.90.31.11:4555: use of closed network connection Sep 11 21:41:12.726683552 (319h32m48.07946869s) [noise=true , address=45.91.116.14:4189 , piece_id=CYLWPQOUM7RBCGRT3G4H4CQWPSPYBCP52LGM27KCUBA6C7XKQVJA, seg_pos={0 0}, upload=281389]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection; write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9136->45.91.116.14:4189: use of closed network connection Sep 11 21:41:12.726703188 (319h32m48.079488327s) [seg_pos={0 0}, upload=281389]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:41:12.743090843 (319h32m48.095875971s) [upload=281389]: Done uploading segments... Sep 11 21:41:12.743111792 (319h32m48.095896921s) [upload=281389]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.019197297s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.001269981s FAIL (2025-09-11 21:40:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001305778s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.....X....] duration: 7.022620072s FAIL (2025-09-11 21:39:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 4.773825809s) [upload=281332]: Starting upload Sep 11 21:37:09.421099020 (319h28m44.773884128s) [upload=281332]: Done starting upload Sep 11 21:37:09.421287414 (319h28m44.774072512s) [upload=281332]: Uploading segments... Sep 11 21:37:09.421327388 (319h28m44.774112496s) [upload=281332]: Got next segment. Inline: false Sep 11 21:37:09.421351864 (319h28m44.774136972s) [upload=281332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:37:09.421355561 (319h28m44.774140659s) [upload=281332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:37:09.445226291 (319h28m44.798011389s) [seg_pos={0 0}, upload=281332]: Begin upload segment... Sep 11 21:37:09.445261377 (319h28m44.798046485s) [seg_pos={0 0}, upload=281332]: Done begin upload segment. Sep 11 21:37:09.445279691 (319h28m44.798064799s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445289319 (319h28m44.798074417s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445345004 (319h28m44.798130112s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445638133 (319h28m44.798423262s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445939669 (319h28m44.798724767s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446226948 (319h28m44.799012046s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446508125 (319h28m44.799293233s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446740561 (319h28m44.799525669s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446966835 (319h28m44.799751933s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447278560 (319h28m44.800063659s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447522007 (319h28m44.800307115s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447811470 (319h28m44.800596567s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448128595 (319h28m44.800913693s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448384665 (319h28m44.801169762s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448651665 (319h28m44.801436773s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.449983584 (319h28m44.802768692s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.533113998 (319h28m44.885899116s) [upload=281332]: Next returned nil segment Sep 11 21:37:09.533128676 (319h28m44.885913784s) [upload=281332]: Waiting for error group managing segments... Sep 11 21:37:09.568123259 (319h28m44.920908367s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.568153555 (319h28m44.920938663s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.571954676 (319h28m44.924739774s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.575213798 (319h28m44.927998896s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.813855839 (319h28m45.166640976s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:33.728395490 (319h29m9.081180619s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.061791094 (319h29m11.414576223s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.872535262 (319h29m12.22532037s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:37.845867471 (319h29m13.1986526s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:39.410403452 (319h29m14.763188541s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: rpc: dial tcp 45.91.116.10:4083: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410601494 (319h29m14.763386622s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410676635 (319h29m14.763461763s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410763157 (319h29m14.763548256s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410803733 (319h29m14.763588831s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:48858->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410841214 (319h29m14.763626311s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection Sep 11 21:37:39.410868996 (319h29m14.763654124s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:60484->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410913058 (319h29m14.763698187s) [seg_pos={0 0}, upload=281332]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:37:39.426472130 (319h29m14.779257238s) [upload=281332]: Done uploading segments... Sep 11 21:37:39.426497006 (319h29m14.779282104s) [upload=281332]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 7.022653645s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.667837468s FAIL (2025-09-11 21:39:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.667876211s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.....X...] duration: 806.348962ms FAIL (2025-09-11 21:39:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 4.773825809s) [upload=281332]: Starting upload Sep 11 21:37:09.421099020 (319h28m44.773884128s) [upload=281332]: Done starting upload Sep 11 21:37:09.421287414 (319h28m44.774072512s) [upload=281332]: Uploading segments... Sep 11 21:37:09.421327388 (319h28m44.774112496s) [upload=281332]: Got next segment. Inline: false Sep 11 21:37:09.421351864 (319h28m44.774136972s) [upload=281332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:37:09.421355561 (319h28m44.774140659s) [upload=281332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:37:09.445226291 (319h28m44.798011389s) [seg_pos={0 0}, upload=281332]: Begin upload segment... Sep 11 21:37:09.445261377 (319h28m44.798046485s) [seg_pos={0 0}, upload=281332]: Done begin upload segment. Sep 11 21:37:09.445279691 (319h28m44.798064799s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445289319 (319h28m44.798074417s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445345004 (319h28m44.798130112s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445638133 (319h28m44.798423262s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445939669 (319h28m44.798724767s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446226948 (319h28m44.799012046s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446508125 (319h28m44.799293233s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446740561 (319h28m44.799525669s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446966835 (319h28m44.799751933s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447278560 (319h28m44.800063659s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447522007 (319h28m44.800307115s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447811470 (319h28m44.800596567s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448128595 (319h28m44.800913693s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448384665 (319h28m44.801169762s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448651665 (319h28m44.801436773s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.449983584 (319h28m44.802768692s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.533113998 (319h28m44.885899116s) [upload=281332]: Next returned nil segment Sep 11 21:37:09.533128676 (319h28m44.885913784s) [upload=281332]: Waiting for error group managing segments... Sep 11 21:37:09.568123259 (319h28m44.920908367s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.568153555 (319h28m44.920938663s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.571954676 (319h28m44.924739774s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.575213798 (319h28m44.927998896s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.813855839 (319h28m45.166640976s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:33.728395490 (319h29m9.081180619s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.061791094 (319h29m11.414576223s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.872535262 (319h29m12.22532037s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:37.845867471 (319h29m13.1986526s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:39.410403452 (319h29m14.763188541s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: rpc: dial tcp 45.91.116.10:4083: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410601494 (319h29m14.763386622s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410676635 (319h29m14.763461763s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410763157 (319h29m14.763548256s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410803733 (319h29m14.763588831s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:48858->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410841214 (319h29m14.763626311s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection Sep 11 21:37:39.410868996 (319h29m14.763654124s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:60484->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410913058 (319h29m14.763698187s) [seg_pos={0 0}, upload=281332]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:37:39.426472130 (319h29m14.779257238s) [upload=281332]: Done uploading segments... Sep 11 21:37:39.426497006 (319h29m14.779282104s) [upload=281332]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 806.370391ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.....X..] duration: 897.616642ms FAIL (2025-09-11 21:38:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 4.773825809s) [upload=281332]: Starting upload Sep 11 21:37:09.421099020 (319h28m44.773884128s) [upload=281332]: Done starting upload Sep 11 21:37:09.421287414 (319h28m44.774072512s) [upload=281332]: Uploading segments... Sep 11 21:37:09.421327388 (319h28m44.774112496s) [upload=281332]: Got next segment. Inline: false Sep 11 21:37:09.421351864 (319h28m44.774136972s) [upload=281332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:37:09.421355561 (319h28m44.774140659s) [upload=281332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:37:09.445226291 (319h28m44.798011389s) [seg_pos={0 0}, upload=281332]: Begin upload segment... Sep 11 21:37:09.445261377 (319h28m44.798046485s) [seg_pos={0 0}, upload=281332]: Done begin upload segment. Sep 11 21:37:09.445279691 (319h28m44.798064799s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445289319 (319h28m44.798074417s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445345004 (319h28m44.798130112s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445638133 (319h28m44.798423262s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445939669 (319h28m44.798724767s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446226948 (319h28m44.799012046s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446508125 (319h28m44.799293233s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446740561 (319h28m44.799525669s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446966835 (319h28m44.799751933s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447278560 (319h28m44.800063659s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447522007 (319h28m44.800307115s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447811470 (319h28m44.800596567s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448128595 (319h28m44.800913693s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448384665 (319h28m44.801169762s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448651665 (319h28m44.801436773s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.449983584 (319h28m44.802768692s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.533113998 (319h28m44.885899116s) [upload=281332]: Next returned nil segment Sep 11 21:37:09.533128676 (319h28m44.885913784s) [upload=281332]: Waiting for error group managing segments... Sep 11 21:37:09.568123259 (319h28m44.920908367s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.568153555 (319h28m44.920938663s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.571954676 (319h28m44.924739774s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.575213798 (319h28m44.927998896s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.813855839 (319h28m45.166640976s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:33.728395490 (319h29m9.081180619s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.061791094 (319h29m11.414576223s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.872535262 (319h29m12.22532037s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:37.845867471 (319h29m13.1986526s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:39.410403452 (319h29m14.763188541s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: rpc: dial tcp 45.91.116.10:4083: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410601494 (319h29m14.763386622s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410676635 (319h29m14.763461763s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410763157 (319h29m14.763548256s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410803733 (319h29m14.763588831s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:48858->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410841214 (319h29m14.763626311s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection Sep 11 21:37:39.410868996 (319h29m14.763654124s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:60484->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410913058 (319h29m14.763698187s) [seg_pos={0 0}, upload=281332]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:37:39.426472130 (319h29m14.779257238s) [upload=281332]: Done uploading segments... Sep 11 21:37:39.426497006 (319h29m14.779282104s) [upload=281332]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 897.655025ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.473699888s FAIL (2025-09-11 21:38:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.473728762s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 925.317417ms FAIL (2025-09-11 21:38:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 925.359254ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration: 3.600198117s FAIL (2025-09-11 21:37:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 4.773825809s) [upload=281332]: Starting upload Sep 11 21:37:09.421099020 (319h28m44.773884128s) [upload=281332]: Done starting upload Sep 11 21:37:09.421287414 (319h28m44.774072512s) [upload=281332]: Uploading segments... Sep 11 21:37:09.421327388 (319h28m44.774112496s) [upload=281332]: Got next segment. Inline: false Sep 11 21:37:09.421351864 (319h28m44.774136972s) [upload=281332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:37:09.421355561 (319h28m44.774140659s) [upload=281332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:37:09.445226291 (319h28m44.798011389s) [seg_pos={0 0}, upload=281332]: Begin upload segment... Sep 11 21:37:09.445261377 (319h28m44.798046485s) [seg_pos={0 0}, upload=281332]: Done begin upload segment. Sep 11 21:37:09.445279691 (319h28m44.798064799s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445289319 (319h28m44.798074417s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445345004 (319h28m44.798130112s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445638133 (319h28m44.798423262s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445939669 (319h28m44.798724767s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446226948 (319h28m44.799012046s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446508125 (319h28m44.799293233s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446740561 (319h28m44.799525669s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446966835 (319h28m44.799751933s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447278560 (319h28m44.800063659s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447522007 (319h28m44.800307115s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447811470 (319h28m44.800596567s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448128595 (319h28m44.800913693s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448384665 (319h28m44.801169762s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448651665 (319h28m44.801436773s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.449983584 (319h28m44.802768692s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.533113998 (319h28m44.885899116s) [upload=281332]: Next returned nil segment Sep 11 21:37:09.533128676 (319h28m44.885913784s) [upload=281332]: Waiting for error group managing segments... Sep 11 21:37:09.568123259 (319h28m44.920908367s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.568153555 (319h28m44.920938663s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.571954676 (319h28m44.924739774s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.575213798 (319h28m44.927998896s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.813855839 (319h28m45.166640976s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:33.728395490 (319h29m9.081180619s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.061791094 (319h29m11.414576223s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.872535262 (319h29m12.22532037s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:37.845867471 (319h29m13.1986526s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:39.410403452 (319h29m14.763188541s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: rpc: dial tcp 45.91.116.10:4083: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410601494 (319h29m14.763386622s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410676635 (319h29m14.763461763s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410763157 (319h29m14.763548256s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410803733 (319h29m14.763588831s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:48858->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410841214 (319h29m14.763626311s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection Sep 11 21:37:39.410868996 (319h29m14.763654124s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:60484->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410913058 (319h29m14.763698187s) [seg_pos={0 0}, upload=281332]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:37:39.426472130 (319h29m14.779257238s) [upload=281332]: Done uploading segments... Sep 11 21:37:39.426497006 (319h29m14.779282104s) [upload=281332]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 3.600229595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration: 30.017423401s FAIL (2025-09-11 21:37:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 4.773825809s) [upload=281332]: Starting upload Sep 11 21:37:09.421099020 (319h28m44.773884128s) [upload=281332]: Done starting upload Sep 11 21:37:09.421287414 (319h28m44.774072512s) [upload=281332]: Uploading segments... Sep 11 21:37:09.421327388 (319h28m44.774112496s) [upload=281332]: Got next segment. Inline: false Sep 11 21:37:09.421351864 (319h28m44.774136972s) [upload=281332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:37:09.421355561 (319h28m44.774140659s) [upload=281332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:37:09.445226291 (319h28m44.798011389s) [seg_pos={0 0}, upload=281332]: Begin upload segment... Sep 11 21:37:09.445261377 (319h28m44.798046485s) [seg_pos={0 0}, upload=281332]: Done begin upload segment. Sep 11 21:37:09.445279691 (319h28m44.798064799s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445289319 (319h28m44.798074417s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445345004 (319h28m44.798130112s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445638133 (319h28m44.798423262s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.445939669 (319h28m44.798724767s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446226948 (319h28m44.799012046s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446508125 (319h28m44.799293233s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446740561 (319h28m44.799525669s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.446966835 (319h28m44.799751933s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447278560 (319h28m44.800063659s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447522007 (319h28m44.800307115s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.447811470 (319h28m44.800596567s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448128595 (319h28m44.800913693s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448384665 (319h28m44.801169762s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.448651665 (319h28m44.801436773s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.449983584 (319h28m44.802768692s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Uploading piece... Sep 11 21:37:09.533113998 (319h28m44.885899116s) [upload=281332]: Next returned nil segment Sep 11 21:37:09.533128676 (319h28m44.885913784s) [upload=281332]: Waiting for error group managing segments... Sep 11 21:37:09.568123259 (319h28m44.920908367s) [noise=true , address=145.79.227.6:4258 , piece_id=CQA7IT2OQJBEVB5BVXTLHDJ5XRKFLIVFKRS3NSYG7YYRCI3WQR6A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.568153555 (319h28m44.920938663s) [noise=true , address=145.79.227.16:4911 , piece_id=4DISYMF3773NPO75KCBOGSOLJY2PWTJ3L2DLAPKCHMZZ2ZNXGLOQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.571954676 (319h28m44.924739774s) [noise=true , address=145.79.227.7:4293 , piece_id=BGYJBYFDAQNLLJF5T6F5UDF7EYUPUVMSC6UIMJE4ZP5EPHOOHZLA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.575213798 (319h28m44.927998896s) [noise=true , address=145.79.227.18:4007 , piece_id=WDE4NKOZZT6QTQQIFUKIARULNXOPFHEQCIRQXCRHAGJ3KYM2QJ4Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:09.813855839 (319h28m45.166640976s) [noise=true , address=145.79.227.13:4801 , piece_id=7DQ72DQD7P62HD4FR6ZCMHOC6P5GG27FPUIV6HH3ZUR32JCS6ULQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:33.728395490 (319h29m9.081180619s) [noise=true , address=45.91.116.4:4938 , piece_id=PVL754IFT7PPOW3LWXUEDYMWR34YMIUGZNV75Y5HITJBSX764MZA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.061791094 (319h29m11.414576223s) [noise=true , address=77.90.31.11:4554 , piece_id=SXUJLXHZRZXQQHXHJJZDDVTHUS5NDB6HI4PGTF3LDCLWLUAXPEJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:36.872535262 (319h29m12.22532037s) [noise=true , address=45.91.116.11:4117 , piece_id=5FCPSHWBPL6VYF4H3W2HRSEXAPUVL4DPKERLUINVBC7JVPXLYQTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:37.845867471 (319h29m13.1986526s) [noise=true , address=77.90.31.16:4734 , piece_id=4DRIGRXTMI2KG5QJDHQVUQXTG72VLUCXNXNS5THCDQMHTDPXLMRQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: <nil> Sep 11 21:37:39.410403452 (319h29m14.763188541s) [noise=true , address=45.91.116.10:4083 , piece_id=TBOSPB4D5IY254SNUUEHDC3EY2VWQR53ODJCODZM5BMBY3B3TYGQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: rpc: dial tcp 45.91.116.10:4083: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410601494 (319h29m14.763386622s) [noise=true , address=45.91.116.8:4045 , piece_id=57HBU7HFWFGS7ZLXKUSPIOEULD4PO4LXLNWJGDVO6B35MBKX6HJA, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410676635 (319h29m14.763461763s) [noise=true , address=45.91.116.11:4118 , piece_id=O2IQWZYMMPM5AWUVUHRHDS4DEBHRMP4WXEQ4UQP3XND4BO56CJ3Q, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410763157 (319h29m14.763548256s) [noise=true , address=77.90.31.18:4806 , piece_id=BXCJSZKPMYX6HZQG77NKD7AZ2VKO26VROXUQIBK7HA2AGXL4USFQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410803733 (319h29m14.763588831s) [noise=true , address=77.90.31.14:4661 , piece_id=RYXUDK2ZC5JH3KFAF646DSSVT25S4PVLOSMJZLTELQLQA3HKVMNQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 5.161.89.252:48858->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410841214 (319h29m14.763626311s) [noise=true , address=45.91.116.6:4975 , piece_id=HKC67JKAXDFASQ6DUNKVDFVZXDZ3PMT2WFNB3PVFLKLQ55QJC63A, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:14114->45.91.116.6:4975: use of closed network connection Sep 11 21:37:39.410868996 (319h29m14.763654124s) [noise=true , address=45.91.116.12:4227 , piece_id=WT2ZOEMDL32PIXAFFR5FNPZGAPINLPORY6ERVLLXD3YC5IJWNOTQ, seg_pos={0 0}, upload=281332]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 5.161.89.252:60484->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:37:39.410913058 (319h29m14.763698187s) [seg_pos={0 0}, upload=281332]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 11 21:37:39.426472130 (319h29m14.779257238s) [upload=281332]: Done uploading segments... Sep 11 21:37:39.426497006 (319h29m14.779282104s) [upload=281332]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.017438629s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00077055s FAIL (2025-09-11 21:37:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000788645s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 524.185846ms FAIL (2025-09-11 21:36:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 11 21:31:49.763568967 (319h23m25.116354095s) [upload=281259]: Starting upload Sep 11 21:31:49.763661109 (319h23m25.116446238s) [upload=281259]: Done starting upload Sep 11 21:31:49.763735078 (319h23m25.116520176s) [upload=281259]: Uploading segments... Sep 11 21:31:49.764595973 (319h23m25.117381101s) [upload=281259]: Got next segment. Inline: false Sep 11 21:31:49.764753939 (319h23m25.117539078s) [upload=281259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:31:49.764785568 (319h23m25.117570697s) [upload=281259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:31:49.787573491 (319h23m25.140358588s) [seg_pos={0 0}, upload=281259]: Begin upload segment... Sep 11 21:31:49.787609137 (319h23m25.140394245s) [seg_pos={0 0}, upload=281259]: Done begin upload segment. Sep 11 21:31:49.787629265 (319h23m25.140414373s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787712080 (319h23m25.140497208s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787978440 (319h23m25.140763548s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788203001 (319h23m25.140988129s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788230202 (319h23m25.14101531s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788479650 (319h23m25.141264748s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788551514 (319h23m25.141336622s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788762580 (319h23m25.141547679s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788773381 (319h23m25.141558489s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788993453 (319h23m25.141778552s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789001810 (319h23m25.141786907s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789249133 (319h23m25.142034241s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789648823 (319h23m25.142433931s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789898982 (319h23m25.14268409s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791044811 (319h23m25.143829909s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791467644 (319h23m25.144252752s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:50.065153534 (319h23m25.417938642s) [upload=281259]: Next returned nil segment Sep 11 21:31:50.065168852 (319h23m25.41795396s) [upload=281259]: Waiting for error group managing segments... Sep 11 21:31:50.127796859 (319h23m25.480581947s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.179702554 (319h23m25.532487653s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.603089075 (319h23m25.955874173s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.017998214 (319h23m45.370783342s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.398533553 (319h23m45.751318661s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.360859104 (319h23m47.713644212s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.777328662 (319h23m48.13011379s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.106146678 (319h23m48.458931807s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.820440179 (319h23m49.173225307s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.097983724 (319h23m49.450768851s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.674987152 (319h23m50.027772249s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:18.383992111 (319h23m53.736777209s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:19.746724617 (319h23m55.099509725s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection Sep 11 21:32:19.746752309 (319h23m55.099537417s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection Sep 11 21:32:19.746775172 (319h23m55.09956028s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 5.161.89.252:9194->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746798015 (319h23m55.099583123s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 5.161.89.252:9906->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746826989 (319h23m55.099612098s) [seg_pos={0 0}, upload=281259]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 11 21:32:19.767533767 (319h23m55.120318895s) [upload=281259]: Done uploading segments... Sep 11 21:32:19.767554436 (319h23m55.120339564s) [upload=281259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 524.226382ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 828.059605ms FAIL (2025-09-11 21:35:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 11 21:31:49.763568967 (319h23m25.116354095s) [upload=281259]: Starting upload Sep 11 21:31:49.763661109 (319h23m25.116446238s) [upload=281259]: Done starting upload Sep 11 21:31:49.763735078 (319h23m25.116520176s) [upload=281259]: Uploading segments... Sep 11 21:31:49.764595973 (319h23m25.117381101s) [upload=281259]: Got next segment. Inline: false Sep 11 21:31:49.764753939 (319h23m25.117539078s) [upload=281259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:31:49.764785568 (319h23m25.117570697s) [upload=281259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:31:49.787573491 (319h23m25.140358588s) [seg_pos={0 0}, upload=281259]: Begin upload segment... Sep 11 21:31:49.787609137 (319h23m25.140394245s) [seg_pos={0 0}, upload=281259]: Done begin upload segment. Sep 11 21:31:49.787629265 (319h23m25.140414373s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787712080 (319h23m25.140497208s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787978440 (319h23m25.140763548s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788203001 (319h23m25.140988129s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788230202 (319h23m25.14101531s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788479650 (319h23m25.141264748s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788551514 (319h23m25.141336622s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788762580 (319h23m25.141547679s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788773381 (319h23m25.141558489s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788993453 (319h23m25.141778552s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789001810 (319h23m25.141786907s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789249133 (319h23m25.142034241s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789648823 (319h23m25.142433931s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789898982 (319h23m25.14268409s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791044811 (319h23m25.143829909s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791467644 (319h23m25.144252752s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:50.065153534 (319h23m25.417938642s) [upload=281259]: Next returned nil segment Sep 11 21:31:50.065168852 (319h23m25.41795396s) [upload=281259]: Waiting for error group managing segments... Sep 11 21:31:50.127796859 (319h23m25.480581947s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.179702554 (319h23m25.532487653s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.603089075 (319h23m25.955874173s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.017998214 (319h23m45.370783342s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.398533553 (319h23m45.751318661s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.360859104 (319h23m47.713644212s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.777328662 (319h23m48.13011379s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.106146678 (319h23m48.458931807s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.820440179 (319h23m49.173225307s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.097983724 (319h23m49.450768851s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.674987152 (319h23m50.027772249s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:18.383992111 (319h23m53.736777209s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:19.746724617 (319h23m55.099509725s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection Sep 11 21:32:19.746752309 (319h23m55.099537417s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection Sep 11 21:32:19.746775172 (319h23m55.09956028s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 5.161.89.252:9194->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746798015 (319h23m55.099583123s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 5.161.89.252:9906->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746826989 (319h23m55.099612098s) [seg_pos={0 0}, upload=281259]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 11 21:32:19.767533767 (319h23m55.120318895s) [upload=281259]: Done uploading segments... Sep 11 21:32:19.767554436 (319h23m55.120339564s) [upload=281259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 828.082066ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 434.985106ms FAIL (2025-09-11 21:35:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 11 21:31:49.763568967 (319h23m25.116354095s) [upload=281259]: Starting upload Sep 11 21:31:49.763661109 (319h23m25.116446238s) [upload=281259]: Done starting upload Sep 11 21:31:49.763735078 (319h23m25.116520176s) [upload=281259]: Uploading segments... Sep 11 21:31:49.764595973 (319h23m25.117381101s) [upload=281259]: Got next segment. Inline: false Sep 11 21:31:49.764753939 (319h23m25.117539078s) [upload=281259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:31:49.764785568 (319h23m25.117570697s) [upload=281259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:31:49.787573491 (319h23m25.140358588s) [seg_pos={0 0}, upload=281259]: Begin upload segment... Sep 11 21:31:49.787609137 (319h23m25.140394245s) [seg_pos={0 0}, upload=281259]: Done begin upload segment. Sep 11 21:31:49.787629265 (319h23m25.140414373s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787712080 (319h23m25.140497208s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787978440 (319h23m25.140763548s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788203001 (319h23m25.140988129s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788230202 (319h23m25.14101531s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788479650 (319h23m25.141264748s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788551514 (319h23m25.141336622s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788762580 (319h23m25.141547679s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788773381 (319h23m25.141558489s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788993453 (319h23m25.141778552s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789001810 (319h23m25.141786907s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789249133 (319h23m25.142034241s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789648823 (319h23m25.142433931s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789898982 (319h23m25.14268409s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791044811 (319h23m25.143829909s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791467644 (319h23m25.144252752s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:50.065153534 (319h23m25.417938642s) [upload=281259]: Next returned nil segment Sep 11 21:31:50.065168852 (319h23m25.41795396s) [upload=281259]: Waiting for error group managing segments... Sep 11 21:31:50.127796859 (319h23m25.480581947s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.179702554 (319h23m25.532487653s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.603089075 (319h23m25.955874173s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.017998214 (319h23m45.370783342s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.398533553 (319h23m45.751318661s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.360859104 (319h23m47.713644212s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.777328662 (319h23m48.13011379s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.106146678 (319h23m48.458931807s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.820440179 (319h23m49.173225307s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.097983724 (319h23m49.450768851s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.674987152 (319h23m50.027772249s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:18.383992111 (319h23m53.736777209s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:19.746724617 (319h23m55.099509725s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection Sep 11 21:32:19.746752309 (319h23m55.099537417s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection Sep 11 21:32:19.746775172 (319h23m55.09956028s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 5.161.89.252:9194->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746798015 (319h23m55.099583123s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 5.161.89.252:9906->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746826989 (319h23m55.099612098s) [seg_pos={0 0}, upload=281259]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 11 21:32:19.767533767 (319h23m55.120318895s) [upload=281259]: Done uploading segments... Sep 11 21:32:19.767554436 (319h23m55.120339564s) [upload=281259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 435.013279ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.277867013s FAIL (2025-09-11 21:34:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 11 21:31:49.763568967 (319h23m25.116354095s) [upload=281259]: Starting upload Sep 11 21:31:49.763661109 (319h23m25.116446238s) [upload=281259]: Done starting upload Sep 11 21:31:49.763735078 (319h23m25.116520176s) [upload=281259]: Uploading segments... Sep 11 21:31:49.764595973 (319h23m25.117381101s) [upload=281259]: Got next segment. Inline: false Sep 11 21:31:49.764753939 (319h23m25.117539078s) [upload=281259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:31:49.764785568 (319h23m25.117570697s) [upload=281259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:31:49.787573491 (319h23m25.140358588s) [seg_pos={0 0}, upload=281259]: Begin upload segment... Sep 11 21:31:49.787609137 (319h23m25.140394245s) [seg_pos={0 0}, upload=281259]: Done begin upload segment. Sep 11 21:31:49.787629265 (319h23m25.140414373s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787712080 (319h23m25.140497208s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787978440 (319h23m25.140763548s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788203001 (319h23m25.140988129s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788230202 (319h23m25.14101531s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788479650 (319h23m25.141264748s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788551514 (319h23m25.141336622s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788762580 (319h23m25.141547679s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788773381 (319h23m25.141558489s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788993453 (319h23m25.141778552s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789001810 (319h23m25.141786907s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789249133 (319h23m25.142034241s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789648823 (319h23m25.142433931s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789898982 (319h23m25.14268409s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791044811 (319h23m25.143829909s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791467644 (319h23m25.144252752s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:50.065153534 (319h23m25.417938642s) [upload=281259]: Next returned nil segment Sep 11 21:31:50.065168852 (319h23m25.41795396s) [upload=281259]: Waiting for error group managing segments... Sep 11 21:31:50.127796859 (319h23m25.480581947s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.179702554 (319h23m25.532487653s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.603089075 (319h23m25.955874173s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.017998214 (319h23m45.370783342s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.398533553 (319h23m45.751318661s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.360859104 (319h23m47.713644212s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.777328662 (319h23m48.13011379s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.106146678 (319h23m48.458931807s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.820440179 (319h23m49.173225307s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.097983724 (319h23m49.450768851s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.674987152 (319h23m50.027772249s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:18.383992111 (319h23m53.736777209s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:19.746724617 (319h23m55.099509725s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection Sep 11 21:32:19.746752309 (319h23m55.099537417s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection Sep 11 21:32:19.746775172 (319h23m55.09956028s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 5.161.89.252:9194->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746798015 (319h23m55.099583123s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 5.161.89.252:9906->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746826989 (319h23m55.099612098s) [seg_pos={0 0}, upload=281259]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 11 21:32:19.767533767 (319h23m55.120318895s) [upload=281259]: Done uploading segments... Sep 11 21:32:19.767554436 (319h23m55.120339564s) [upload=281259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.277887963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.213359377s FAIL (2025-09-11 21:32:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 11 21:31:49.763568967 (319h23m25.116354095s) [upload=281259]: Starting upload Sep 11 21:31:49.763661109 (319h23m25.116446238s) [upload=281259]: Done starting upload Sep 11 21:31:49.763735078 (319h23m25.116520176s) [upload=281259]: Uploading segments... Sep 11 21:31:49.764595973 (319h23m25.117381101s) [upload=281259]: Got next segment. Inline: false Sep 11 21:31:49.764753939 (319h23m25.117539078s) [upload=281259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:31:49.764785568 (319h23m25.117570697s) [upload=281259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:31:49.787573491 (319h23m25.140358588s) [seg_pos={0 0}, upload=281259]: Begin upload segment... Sep 11 21:31:49.787609137 (319h23m25.140394245s) [seg_pos={0 0}, upload=281259]: Done begin upload segment. Sep 11 21:31:49.787629265 (319h23m25.140414373s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787712080 (319h23m25.140497208s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787978440 (319h23m25.140763548s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788203001 (319h23m25.140988129s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788230202 (319h23m25.14101531s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788479650 (319h23m25.141264748s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788551514 (319h23m25.141336622s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788762580 (319h23m25.141547679s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788773381 (319h23m25.141558489s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788993453 (319h23m25.141778552s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789001810 (319h23m25.141786907s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789249133 (319h23m25.142034241s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789648823 (319h23m25.142433931s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789898982 (319h23m25.14268409s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791044811 (319h23m25.143829909s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791467644 (319h23m25.144252752s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:50.065153534 (319h23m25.417938642s) [upload=281259]: Next returned nil segment Sep 11 21:31:50.065168852 (319h23m25.41795396s) [upload=281259]: Waiting for error group managing segments... Sep 11 21:31:50.127796859 (319h23m25.480581947s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.179702554 (319h23m25.532487653s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.603089075 (319h23m25.955874173s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.017998214 (319h23m45.370783342s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.398533553 (319h23m45.751318661s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.360859104 (319h23m47.713644212s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.777328662 (319h23m48.13011379s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.106146678 (319h23m48.458931807s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.820440179 (319h23m49.173225307s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.097983724 (319h23m49.450768851s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.674987152 (319h23m50.027772249s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:18.383992111 (319h23m53.736777209s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:19.746724617 (319h23m55.099509725s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection Sep 11 21:32:19.746752309 (319h23m55.099537417s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection Sep 11 21:32:19.746775172 (319h23m55.09956028s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 5.161.89.252:9194->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746798015 (319h23m55.099583123s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 5.161.89.252:9906->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746826989 (319h23m55.099612098s) [seg_pos={0 0}, upload=281259]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 11 21:32:19.767533767 (319h23m55.120318895s) [upload=281259]: Done uploading segments... Sep 11 21:32:19.767554436 (319h23m55.120339564s) [upload=281259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.213390706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.022460222s FAIL (2025-09-11 21:32:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 11 21:31:49.763568967 (319h23m25.116354095s) [upload=281259]: Starting upload Sep 11 21:31:49.763661109 (319h23m25.116446238s) [upload=281259]: Done starting upload Sep 11 21:31:49.763735078 (319h23m25.116520176s) [upload=281259]: Uploading segments... Sep 11 21:31:49.764595973 (319h23m25.117381101s) [upload=281259]: Got next segment. Inline: false Sep 11 21:31:49.764753939 (319h23m25.117539078s) [upload=281259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 11 21:31:49.764785568 (319h23m25.117570697s) [upload=281259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 11 21:31:49.787573491 (319h23m25.140358588s) [seg_pos={0 0}, upload=281259]: Begin upload segment... Sep 11 21:31:49.787609137 (319h23m25.140394245s) [seg_pos={0 0}, upload=281259]: Done begin upload segment. Sep 11 21:31:49.787629265 (319h23m25.140414373s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787712080 (319h23m25.140497208s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.787978440 (319h23m25.140763548s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788203001 (319h23m25.140988129s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788230202 (319h23m25.14101531s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788479650 (319h23m25.141264748s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788551514 (319h23m25.141336622s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788762580 (319h23m25.141547679s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788773381 (319h23m25.141558489s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.788993453 (319h23m25.141778552s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789001810 (319h23m25.141786907s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789249133 (319h23m25.142034241s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789648823 (319h23m25.142433931s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.789898982 (319h23m25.14268409s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791044811 (319h23m25.143829909s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:49.791467644 (319h23m25.144252752s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Uploading piece... Sep 11 21:31:50.065153534 (319h23m25.417938642s) [upload=281259]: Next returned nil segment Sep 11 21:31:50.065168852 (319h23m25.41795396s) [upload=281259]: Waiting for error group managing segments... Sep 11 21:31:50.127796859 (319h23m25.480581947s) [noise=true , address=145.79.227.4:4186 , piece_id=65THZFZVPL53JHLVJV44IG3ARATIFSYNPCCMPOGUEBEF7RTVQFAA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.179702554 (319h23m25.532487653s) [noise=true , address=145.79.227.19:4078 , piece_id=PR5W4WT5SSJ5CV6RG7VT4I2CFINQJZH4JZH3CAPGRO6HKIBJONTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:31:50.603089075 (319h23m25.955874173s) [noise=true , address=145.79.227.14:4839 , piece_id=T4ECJ4ZF3LZ6AMY3Z3NTPK6ZS7VTY47MOYJLGQ6UOH33YOXVVTCQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.017998214 (319h23m45.370783342s) [noise=true , address=45.91.116.10:4081 , piece_id=2Q27ACDQXVTCR3ZGI6743H6LYAXIUJT5YSTPJPWN7JNK3HP7DOKQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:10.398533553 (319h23m45.751318661s) [noise=true , address=77.90.31.16:4733 , piece_id=GVK4ZCJSUVQR75L6FVJZTOS2UMESB4FHAUSFD6DNXEWXARVQHW3Q, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.360859104 (319h23m47.713644212s) [noise=true , address=45.91.116.8:4047 , piece_id=2F447NN2YRRB4OPGYTW6C2QFQZLSGSHZI5MTWBOAFOPHMCISUWZA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:12.777328662 (319h23m48.13011379s) [noise=true , address=45.91.116.4:4940 , piece_id=DBKGZXES6E2FRQM24NWNO6KQP5RV3JUCFJQXORLIGOGVUGVKC43A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.106146678 (319h23m48.458931807s) [noise=true , address=77.90.31.15:4699 , piece_id=G2KJHXUQFRWTH6NVQJBARPOUJPAE25VQLTNXVGPDCUPEYNKZNXLA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:13.820440179 (319h23m49.173225307s) [noise=true , address=45.91.116.12:4226 , piece_id=S4OOZMBHUIGDDRNL2CLRXKOXJOWVRT4GAEWSKJ4ANK4TRBW7ZNXA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.097983724 (319h23m49.450768851s) [noise=true , address=45.91.116.7:4010 , piece_id=FRVOFAYXTXWU6QNB6BKDA2KATXZ5DQRZGVVKRT2GHPV6QJJEMEUQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:14.674987152 (319h23m50.027772249s) [noise=true , address=45.91.116.7:4011 , piece_id=HTA2IF3ZPB7LI5SZFW2W3OE4HIS2YSAML4LW34YPLMGWUDMIK4HA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:18.383992111 (319h23m53.736777209s) [noise=true , address=45.91.116.9:4261 , piece_id=XDJUN3E66MV6C4BDVF24MXCHTYYIOECE73OFT6IWGVFKXBKQ2UTQ, seg_pos={0 0}, upload=281259]: Done uploading piece. err: <nil> Sep 11 21:32:19.746724617 (319h23m55.099509725s) [noise=true , address=77.90.31.19:4843 , piece_id=DEDOJTOEGXD6SDMVHPBWI4QMPLDKFKMPBRITPAS7XEL3XCKO527A, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:49350->77.90.31.19:4843: use of closed network connection Sep 11 21:32:19.746752309 (319h23m55.099537417s) [noise=true , address=45.91.116.13:4154 , piece_id=F435GIGXLNAEVQS224DCGREY4IWR7CAM7ROULRNT426AHV2NCPNA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:9106->45.91.116.13:4154: use of closed network connection Sep 11 21:32:19.746775172 (319h23m55.09956028s) [noise=true , address=77.90.31.15:4697 , piece_id=HPZK5SQXBV77RBRY35HEH67AEIT4M2GNP4ANYG67Y7QT2KA7EIPA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 5.161.89.252:9194->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746798015 (319h23m55.099583123s) [noise=true , address=45.91.116.5:4299 , piece_id=FQ7ADLUGRVJCXJ574PU7K2QV2VLALTF5TQYSSLZHWM52HTDWFHFA, seg_pos={0 0}, upload=281259]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 5.161.89.252:9906->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 11 21:32:19.746826989 (319h23m55.099612098s) [seg_pos={0 0}, upload=281259]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 11 21:32:19.767533767 (319h23m55.120318895s) [upload=281259]: Done uploading segments... Sep 11 21:32:19.767554436 (319h23m55.120339564s) [upload=281259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.022567102s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 466.195037ms FAIL (2025-09-11 21:16:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 466.228489ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 785.595524ms FAIL (2025-09-11 21:15:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 785.614229ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 701.105399ms FAIL (2025-09-11 21:13:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 701.129284ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 435.968212ms FAIL (2025-09-11 21:13:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 435.990083ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 624.961204ms FAIL (2025-09-11 21:12:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 624.979027ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 763.620199ms FAIL (2025-09-11 21:11:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 763.641509ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 425.843359ms FAIL (2025-09-11 21:11:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 425.860422ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 517.117466ms FAIL (2025-09-11 21:10:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 517.135339ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)