Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 110692 reports (Page 141 of 2214)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X..X.X....X.X...............] duration: 359.963896ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:46:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 359.985045ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................XX.XX..............................] duration: 445.803946ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:45:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.824825ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.013762319s trace: https://jaeger.dev.storj.io/trace/c07bec575cab50d FAIL (2025-12-10 23:45:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (128h15m38.474854813s) [upload=112592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:44:50.795426092 (128h15m38.492994398s) [seg_pos={0 0}, upload=112592]: Begin upload segment... Dec 10 23:44:50.795462991 (128h15m38.493031307s) [seg_pos={0 0}, upload=112592]: Done begin upload segment. Dec 10 23:44:50.795535326 (128h15m38.493103633s) [noise=true , address=77.90.31.12:4619 , piece_id=EC4HRH46BYPJ3ENZNISETQW2P3UBF5OQBBFEME5PE74PDEJSIBLQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.795548301 (128h15m38.493116617s) [noise=true , address=145.79.227.30:4796 , piece_id=QOOEKGSR7M6GMLAPGW7FHT6SIAMHCZ3HRZ6TJQFYGAM744UY2AXQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.796024951 (128h15m38.493593267s) [noise=true , address=45.91.116.7:4017 , piece_id=ZQ7WAOBZFVTJMCVUEHITKVWFYP5AEEALJKLWZIE7W5TYMCYQ7TEA, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.796374935 (128h15m38.493943251s) [noise=true , address=145.79.227.42:4150 , piece_id=ENPYW5OW6GB2JX3EUQFOO35NQX6RZXJKXEIN4AW2SHM2SC6KG2DQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.796463820 (128h15m38.494032127s) [noise=true , address=77.90.31.15:4724 , piece_id=GAI44HDXRDALBAMTMT2VNZBRIKB5P6SRRSBZWEAYP3AKYNIAQHXA, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.796675766 (128h15m38.494244082s) [noise=true , address=45.91.116.4:4944 , piece_id=WBSPLBIXCO34ONSZG26FEZE6BUDP5PMGB3IAOEKI3JTJKJRO4WKQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.796886621 (128h15m38.494454926s) [noise=true , address=5.231.73.2:4031 , piece_id=HATXM5D27LDHBRKJXVWQPCI24A2OQCHC3QCWNB4INXQBO7MZVOLA, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.796953646 (128h15m38.494521952s) [noise=true , address=145.79.227.24:4616 , piece_id=N5D6PY3ZQUIAKPAHBYR5LB32WETXOP3HEJZWROW3D4GMGI7QHI4Q, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.797298801 (128h15m38.494867107s) [noise=true , address=77.90.31.13:4635 , piece_id=NMZQOIKI5VCQNBHGYFUPERPLACPSNHCFVQKPNG7DDQJF7UST6EDQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.797310462 (128h15m38.494878779s) [noise=true , address=145.79.227.34:4917 , piece_id=TJOCKKPG37URFIAUTMXPCU5QHX2VPLF5S5P7BRY5CYDAIE56RJWA, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.797594022 (128h15m38.495162328s) [noise=true , address=145.79.227.22:4542 , piece_id=4SWRTSAF3IGIUVDLOJGSKRJ4WWVIXSKDCYLACS5J3FPHC6XIIYFA, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.797697145 (128h15m38.495265461s) [noise=true , address=45.91.116.11:4141 , piece_id=UU3PXN45ID3QBTN47A5G6ONFCLXWX6TX7HW4VF3XGEE6T7YRHYOQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.797841765 (128h15m38.49541007s) [noise=true , address=145.79.227.52:4466 , piece_id=QURSZIVOPT4SUWWXUKU65PQNI7OTUWAHJJ335CQM2I373DL7BLPQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.798024917 (128h15m38.495593233s) [noise=true , address=145.79.227.16:4919 , piece_id=OPNNXM7E77LATGE7IPGTUENVHUMPMCGRUBYDCFFBWAGAI2IXG2PA, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.798098484 (128h15m38.4956668s) [noise=true , address=77.90.31.11:4581 , piece_id=T4NSCYFD22INE4LR4UL2M7CPKQJ6S3YMQTWIVL4BHWMHDXNGYP5Q, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.798357267 (128h15m38.495925583s) [noise=true , address=145.79.227.20:4471 , piece_id=MPPNRFATPTISM5DBCULCSUOXVJBHLP7BOJ6WIL5AJDPUGMXCY3FQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.798428060 (128h15m38.495996366s) [noise=true , address=77.90.31.13:4658 , piece_id=B6ZUBXJSSHE3G7DEXQ3GVA6KIVVXCOUEYC5YT4A4SMYPKOGYMDBA, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.798695620 (128h15m38.496263926s) [noise=true , address=77.90.31.16:4743 , piece_id=LDADMWH7YOHUIL7MZU67ENODYZDHHRGSTM6DK2IOG4NROXNS3KGQ, seg_pos={0 0}, upload=112592]: Uploading piece... Dec 10 23:44:50.876365948 (128h15m38.573934255s) [upload=112592]: Next returned nil segment Dec 10 23:44:50.876375196 (128h15m38.573943502s) [upload=112592]: Waiting for error group managing segments... Dec 10 23:44:50.888442457 (128h15m38.586010764s) [noise=true , address=145.79.227.34:4917 , piece_id=TJOCKKPG37URFIAUTMXPCU5QHX2VPLF5S5P7BRY5CYDAIE56RJWA, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:50.889342930 (128h15m38.586911256s) [noise=true , address=145.79.227.20:4471 , piece_id=MPPNRFATPTISM5DBCULCSUOXVJBHLP7BOJ6WIL5AJDPUGMXCY3FQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:50.895987282 (128h15m38.593555598s) [noise=true , address=145.79.227.24:4616 , piece_id=N5D6PY3ZQUIAKPAHBYR5LB32WETXOP3HEJZWROW3D4GMGI7QHI4Q, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:50.899341938 (128h15m38.596910254s) [noise=true , address=145.79.227.22:4542 , piece_id=4SWRTSAF3IGIUVDLOJGSKRJ4WWVIXSKDCYLACS5J3FPHC6XIIYFA, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:50.902675856 (128h15m38.600244182s) [noise=true , address=45.91.116.11:4141 , piece_id=UU3PXN45ID3QBTN47A5G6ONFCLXWX6TX7HW4VF3XGEE6T7YRHYOQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:50.935693163 (128h15m38.633261499s) [noise=true , address=145.79.227.52:4466 , piece_id=QURSZIVOPT4SUWWXUKU65PQNI7OTUWAHJJ335CQM2I373DL7BLPQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:50.940626328 (128h15m38.638194664s) [noise=true , address=45.91.116.4:4944 , piece_id=WBSPLBIXCO34ONSZG26FEZE6BUDP5PMGB3IAOEKI3JTJKJRO4WKQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:50.978876319 (128h15m38.676444635s) [noise=true , address=145.79.227.42:4150 , piece_id=ENPYW5OW6GB2JX3EUQFOO35NQX6RZXJKXEIN4AW2SHM2SC6KG2DQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:51.003875377 (128h15m38.701443703s) [noise=true , address=5.231.73.2:4031 , piece_id=HATXM5D27LDHBRKJXVWQPCI24A2OQCHC3QCWNB4INXQBO7MZVOLA, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:51.085732897 (128h15m38.783301204s) [noise=true , address=45.91.116.7:4017 , piece_id=ZQ7WAOBZFVTJMCVUEHITKVWFYP5AEEALJKLWZIE7W5TYMCYQ7TEA, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:51.871396344 (128h15m39.56896465s) [noise=true , address=145.79.227.30:4796 , piece_id=QOOEKGSR7M6GMLAPGW7FHT6SIAMHCZ3HRZ6TJQFYGAM744UY2AXQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:44:51.932089379 (128h15m39.629657695s) [noise=true , address=145.79.227.16:4919 , piece_id=OPNNXM7E77LATGE7IPGTUENVHUMPMCGRUBYDCFFBWAGAI2IXG2PA, seg_pos={0 0}, upload=112592]: Done uploading piece. err: <nil> Dec 10 23:45:50.760298675 (128h16m38.457866981s) [noise=true , address=77.90.31.15:4724 , piece_id=GAI44HDXRDALBAMTMT2VNZBRIKB5P6SRRSBZWEAYP3AKYNIAQHXA, seg_pos={0 0}, upload=112592]: Done uploading piece. err: ecclient: upload failed (node:12YEcohZBsQSm45orPuT8np4EGgXQaWAM8rCsmUfA1PfNubwLmo, address:77.90.31.15:4724): protocol: write tcp 5.161.89.252:14866->77.90.31.15:4724: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:50.760366792 (128h16m38.457935108s) [noise=true , address=77.90.31.16:4743 , piece_id=LDADMWH7YOHUIL7MZU67ENODYZDHHRGSTM6DK2IOG4NROXNS3KGQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: rpc: dial tcp 77.90.31.16:4743: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:50.760508146 (128h16m38.458076452s) [noise=true , address=77.90.31.13:4635 , piece_id=NMZQOIKI5VCQNBHGYFUPERPLACPSNHCFVQKPNG7DDQJF7UST6EDQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:50.760665931 (128h16m38.458234237s) [noise=true , address=77.90.31.12:4619 , piece_id=EC4HRH46BYPJ3ENZNISETQW2P3UBF5OQBBFEME5PE74PDEJSIBLQ, seg_pos={0 0}, upload=112592]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:50.760834366 (128h16m38.458402671s) [noise=true , address=77.90.31.11:4581 , piece_id=T4NSCYFD22INE4LR4UL2M7CPKQJ6S3YMQTWIVL4BHWMHDXNGYP5Q, seg_pos={0 0}, upload=112592]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: rpc: dial tcp 77.90.31.11:4581: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:50.760880512 (128h16m38.458448818s) [noise=true , address=77.90.31.13:4658 , piece_id=B6ZUBXJSSHE3G7DEXQ3GVA6KIVVXCOUEYC5YT4A4SMYPKOGYMDBA, seg_pos={0 0}, upload=112592]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: write tcp 5.161.89.252:20624->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:45:50.760903535 (128h16m38.45847184s) [seg_pos={0 0}, upload=112592]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:45:50.772905084 (128h16m38.4704734s) [upload=112592]: Done uploading segments... Dec 10 23:45:50.772933938 (128h16m38.470502244s) [upload=112592]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.013804919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 426.275056ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-10 23:45:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 426.292278ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................XX.XX.............................] duration: 456.747496ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:45:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.764148ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................XXX.............................] duration: 605.392647ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:45:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.416072ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................XX.XX............................] duration: 536.381144ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:45:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 536.432661ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 463.802706ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-10 23:45:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.820508ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................XXX............................] duration: 577.011921ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:45:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 577.028021ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X..X.X....X.X..............] duration: 327.609057ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:45:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 327.623664ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................XX.XX...........................] duration: 529.516381ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:44:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 529.651384ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.016999688s trace: https://jaeger.dev.storj.io/trace/eb7d1accf36af93 FAIL (2025-12-10 23:44:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== JEBRTCTN5YFXB6Q, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.734787086 (128h14m38.432355423s) [noise=true , address=45.91.116.4:4967 , piece_id=CB6LGKZW3M3IEQS4BITGHCJINB3NAKVL27SJA3777ABESSYJ4IDQ, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.735487795 (128h14m38.433056141s) [noise=true , address=145.79.227.46:4617 , piece_id=QCKGLZ5LO2BJJJBIQEPH5CFTQUBXWFCPMTBWOIB7DT5L7OIOVSJA, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.736236674 (128h14m38.433805021s) [noise=true , address=77.90.31.21:4891 , piece_id=DD4EUM2D6HQRUOKFVTMV6GSK5F2AQZDDAOZ4GTDOOD3SSYP4ZUXA, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.736916935 (128h14m38.434485291s) [noise=true , address=77.90.31.16:4768 , piece_id=2BS3ZI3P36BDDFARVMBGYSY2TA2W7JFKQ2PQUH5DXVGHHHXYKT4A, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.737594832 (128h14m38.435163178s) [noise=true , address=77.90.31.19:4860 , piece_id=VM4332MN7Q2I3BBN37WKHTAN5LVJDIMT3BCQY3SFGWLNX7JOMSTQ, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.738252009 (128h14m38.435820345s) [noise=true , address=77.90.31.19:4861 , piece_id=RX65IRVU75MTRED7GNMSDMTA7HGEGEM2NO7HBBPX5LFCKZ2PSZIA, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.738968628 (128h14m38.436536994s) [noise=true , address=45.91.116.9:4294 , piece_id=PVJTL5SKQ6OKYG66IYAEIGJ23M6XGAHXOBGPXG3DJRJHJAKDSQXA, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.739670469 (128h14m38.437238825s) [noise=true , address=77.90.31.17:4785 , piece_id=77ZQ75ID3762YO5NYNDY3YOSPMLMQYI5JRQZMY4K2KDTKPUQX2KQ, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.740409329 (128h14m38.437977665s) [noise=true , address=145.79.227.30:4784 , piece_id=ZIUJ3XUB5FKUVBPL4RSOVTMTH7XSY3CG7RAWALBEHSRICTIBJQ2A, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.741460242 (128h14m38.439028598s) [noise=true , address=77.90.31.18:4840 , piece_id=K5TGHWDTJQTMQWEDRY77NCL6IPNCGH2YINRIE6EJUHWWAEFFSLZA, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.742304849 (128h14m38.439873185s) [noise=true , address=145.79.227.30:4786 , piece_id=N4CJK76BVOSYV5N5IEKDQP3CUJQSGAWJRIE35Y53PMJYHXJ57V4Q, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.743095868 (128h14m38.440664203s) [noise=true , address=45.91.116.5:4314 , piece_id=VFS6KKDOAAEMMIOAKQXBYVKPV2LFUSGPUPI5WUJC6QTDRQJA6P2A, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.743890271 (128h14m38.441458607s) [noise=true , address=145.79.227.21:4486 , piece_id=LRLRAP7QZHJ7RF7T4MJUUETG5T3OHNI2YZFMGKODJETFQNN3GVFQ, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.744740540 (128h14m38.442308875s) [noise=true , address=77.90.31.14:4678 , piece_id=I3PSTG2BTTH6J2JEN5J3MFTJMJGMSSXBGP4SURTWBYWBXHXIVNEQ, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.745553648 (128h14m38.443121984s) [noise=true , address=145.79.227.51:4421 , piece_id=CID6I4VVA3VEHQW66VOXARVYCYHG5FR2FM2H3SX6ENZULHQOCQBA, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.747182221 (128h14m38.444750567s) [noise=true , address=64.49.13.137:4497 , piece_id=OB64LASVOQFPFAZJF4R3TTGTFG2D5SY3JQS4ASYWW5YMQV556RLQ, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.755207702 (128h14m38.452776028s) [noise=true , address=77.90.31.13:4646 , piece_id=UXYZLBMVIBQMHS4SAVS6BDZDSZSDTYVWEYJ3XYULT4VQLH3FJ56A, seg_pos={0 0}, upload=112576]: Uploading piece... Dec 10 23:43:50.832973659 (128h14m38.530541975s) [upload=112576]: Next returned nil segment Dec 10 23:43:50.832982446 (128h14m38.530550763s) [upload=112576]: Waiting for error group managing segments... Dec 10 23:43:50.842931952 (128h14m38.540500257s) [noise=true , address=145.79.227.51:4421 , piece_id=CID6I4VVA3VEHQW66VOXARVYCYHG5FR2FM2H3SX6ENZULHQOCQBA, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:50.843268020 (128h14m38.540836326s) [noise=true , address=145.79.227.46:4617 , piece_id=QCKGLZ5LO2BJJJBIQEPH5CFTQUBXWFCPMTBWOIB7DT5L7OIOVSJA, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:50.843800895 (128h14m38.541369201s) [noise=true , address=45.91.116.9:4294 , piece_id=PVJTL5SKQ6OKYG66IYAEIGJ23M6XGAHXOBGPXG3DJRJHJAKDSQXA, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:50.844127245 (128h14m38.541695561s) [noise=true , address=145.79.227.21:4486 , piece_id=LRLRAP7QZHJ7RF7T4MJUUETG5T3OHNI2YZFMGKODJETFQNN3GVFQ, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:50.856100431 (128h14m38.553668747s) [noise=true , address=45.91.116.4:4967 , piece_id=CB6LGKZW3M3IEQS4BITGHCJINB3NAKVL27SJA3777ABESSYJ4IDQ, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:50.856723324 (128h14m38.55429165s) [noise=true , address=45.91.116.5:4314 , piece_id=VFS6KKDOAAEMMIOAKQXBYVKPV2LFUSGPUPI5WUJC6QTDRQJA6P2A, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:50.860755036 (128h14m38.558323352s) [noise=true , address=45.91.116.10:4115 , piece_id=SCSSZQFW5BYO2IEEYWC37WMD2DRZZHAHHNT2FJEBRTCTN5YFXB6Q, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:51.325021269 (128h14m39.022589575s) [noise=true , address=64.49.13.137:4497 , piece_id=OB64LASVOQFPFAZJF4R3TTGTFG2D5SY3JQS4ASYWW5YMQV556RLQ, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:51.803983172 (128h14m39.501551519s) [noise=true , address=145.79.227.30:4784 , piece_id=ZIUJ3XUB5FKUVBPL4RSOVTMTH7XSY3CG7RAWALBEHSRICTIBJQ2A, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:43:51.815751558 (128h14m39.513319893s) [noise=true , address=145.79.227.30:4786 , piece_id=N4CJK76BVOSYV5N5IEKDQP3CUJQSGAWJRIE35Y53PMJYHXJ57V4Q, seg_pos={0 0}, upload=112576]: Done uploading piece. err: <nil> Dec 10 23:44:50.688757636 (128h15m38.386325941s) [noise=true , address=77.90.31.16:4768 , piece_id=2BS3ZI3P36BDDFARVMBGYSY2TA2W7JFKQ2PQUH5DXVGHHHXYKT4A, seg_pos={0 0}, upload=112576]: Done uploading piece. err: ecclient: upload failed (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF, address:77.90.31.16:4768): protocol: write tcp 5.161.89.252:45852->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:50.688769707 (128h15m38.386338013s) [noise=true , address=77.90.31.17:4785 , piece_id=77ZQ75ID3762YO5NYNDY3YOSPMLMQYI5JRQZMY4K2KDTKPUQX2KQ, seg_pos={0 0}, upload=112576]: Done uploading piece. err: ecclient: upload failed (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN, address:77.90.31.17:4785): protocol: rpc: dial tcp 77.90.31.17:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:50.688911642 (128h15m38.386479948s) [noise=true , address=77.90.31.19:4861 , piece_id=RX65IRVU75MTRED7GNMSDMTA7HGEGEM2NO7HBBPX5LFCKZ2PSZIA, seg_pos={0 0}, upload=112576]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: rpc: dial tcp 77.90.31.19:4861: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:50.689023733 (128h15m38.386592049s) [noise=true , address=77.90.31.13:4646 , piece_id=UXYZLBMVIBQMHS4SAVS6BDZDSZSDTYVWEYJ3XYULT4VQLH3FJ56A, seg_pos={0 0}, upload=112576]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 5.161.89.252:54586->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:50.689112878 (128h15m38.386681185s) [noise=true , address=77.90.31.18:4840 , piece_id=K5TGHWDTJQTMQWEDRY77NCL6IPNCGH2YINRIE6EJUHWWAEFFSLZA, seg_pos={0 0}, upload=112576]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: rpc: dial tcp 77.90.31.18:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:50.689276143 (128h15m38.38684444s) [noise=true , address=77.90.31.19:4860 , piece_id=VM4332MN7Q2I3BBN37WKHTAN5LVJDIMT3BCQY3SFGWLNX7JOMSTQ, seg_pos={0 0}, upload=112576]: Done uploading piece. err: ecclient: upload failed (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, address:77.90.31.19:4860): protocol: rpc: dial tcp 77.90.31.19:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:50.689426424 (128h15m38.386994731s) [noise=true , address=77.90.31.21:4891 , piece_id=DD4EUM2D6HQRUOKFVTMV6GSK5F2AQZDDAOZ4GTDOOD3SSYP4ZUXA, seg_pos={0 0}, upload=112576]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:50.689464134 (128h15m38.38703243s) [noise=true , address=77.90.31.14:4678 , piece_id=I3PSTG2BTTH6J2JEN5J3MFTJMJGMSSXBGP4SURTWBYWBXHXIVNEQ, seg_pos={0 0}, upload=112576]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:44:50.689498419 (128h15m38.387066724s) [seg_pos={0 0}, upload=112576]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:44:50.704491695 (128h15m38.402060011s) [upload=112576]: Done uploading segments... Dec 10 23:44:50.704524717 (128h15m38.402093033s) [upload=112576]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.017010197s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration: 582.69591ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:44:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.736866ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................XX.XX..........................] duration: 485.611224ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:44:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.633726ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration: 616.059795ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:44:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 616.079152ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................XX.XX.........................] duration: 463.25369ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:43:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 463.298683ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 431.334717ms trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-10 23:43:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.349775ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.017722377s trace: https://jaeger.dev.storj.io/trace/3e828cdd3453d1a6 FAIL (2025-12-10 23:43:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== OV7PAI4OGWBEB26Q, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.660487152 (128h13m38.358055459s) [noise=true , address=45.91.116.9:4291 , piece_id=XKBT7V2KFCUJXSKCKNM4PT3FF7EUDEKA3IZWARZQ4UOT2G2TWQWA, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.660920081 (128h13m38.358488397s) [noise=true , address=77.90.31.13:4648 , piece_id=BEXDN23DVUPXHFUMYZVSH43WSWVLGAUH6WMWHUL5DBUSCMVMYO6Q, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.661276767 (128h13m38.358845083s) [noise=true , address=77.90.31.21:4897 , piece_id=ZQ6S4ZEVZMS4MI6YTLMDELPM64MUETF5O543FXQXIBHGMG72ZPWA, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.661652588 (128h13m38.359220904s) [noise=true , address=77.90.31.12:4601 , piece_id=SLIGIX7PLI4YW6PEPAWAAOO2X62N2V2WSELY4HZOVEL22YVSIG5A, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.662032790 (128h13m38.359601106s) [noise=true , address=5.231.73.4:4031 , piece_id=KNIZEUUIPZGMWEEMKZGPVH3NEIOSJMTBFOA6GAFFIEPEHSCFWR6A, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.662421516 (128h13m38.359989852s) [noise=true , address=45.91.116.10:4114 , piece_id=4Q2LXQQWV66YRI5LVHALF2OFTV7OPFHQVEJRQTFC3DFQSHFXDABQ, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.662829297 (128h13m38.360397643s) [noise=true , address=45.91.116.5:4331 , piece_id=75CMZZD7IAVEH672OX2JSB5QJRTDIXYZIPN6SMY3YDTX2J3IU4JQ, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.663178480 (128h13m38.360746786s) [noise=true , address=64.49.13.138:4488 , piece_id=B32VA6WJFMYZTZ22Z565MKAMXZMUHOECMJF2NOWHCV6RNDOYDFJA, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.663447773 (128h13m38.361016089s) [noise=true , address=45.91.116.11:4134 , piece_id=6M24N33DZRSOQS6ECCW2N3RYRUUPGJKQEPJEAGCVZDFXC2GGD7QA, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.663764063 (128h13m38.361332379s) [noise=true , address=77.90.31.10:4546 , piece_id=WWWJSUNAIW7DYCLQF4DLXAT6LSJ43WRTHN4L2D2VGSVW7YVWITAQ, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.663770405 (128h13m38.361338721s) [noise=true , address=77.90.31.16:4762 , piece_id=HIILELXRPQJUJICKFSPZA2CDAPLEJUFPAJATOGJ6Y43T3ENYOTKQ, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.664134125 (128h13m38.361702431s) [noise=true , address=45.91.116.14:4198 , piece_id=2SR76LLNBZN4ZSBMNCVGYR6N4BAM6PG5NK2MCR3YIJ2GBG235K5Q, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.664436280 (128h13m38.362004625s) [noise=true , address=77.90.31.11:4586 , piece_id=QWD3OSPAGIWXGOGGNZ52M73FIN4A5ELZ6OAMNTTELDBRK6UAXOKQ, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.664775523 (128h13m38.362343829s) [noise=true , address=145.79.227.47:4296 , piece_id=KKDVH3AYSCCOCPRAHJRGBW4HANQWGZTGDISA3F3KOFFWMR7FLJJQ, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.665266410 (128h13m38.362834726s) [noise=true , address=77.90.31.15:4722 , piece_id=F2VKK5S4SSHFWTTG53ZM3SFJNYG7ATWW6X52F7NAEWJR7W2AL4SA, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.665686595 (128h13m38.363254911s) [noise=true , address=5.231.73.4:4034 , piece_id=KMX6JS77IS6EBJLPD326LJYNUKC4XTJOMQ24KTE65F42WMXSTVXA, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.666021710 (128h13m38.363590016s) [noise=true , address=77.90.31.14:4693 , piece_id=K65PGTJX7GKDCAGFDTBRLV3Z4KIZYVXOKVSYGUXQTQY2ABU4VSOA, seg_pos={0 0}, upload=112562]: Uploading piece... Dec 10 23:42:50.727464306 (128h13m38.425032613s) [upload=112562]: Next returned nil segment Dec 10 23:42:50.727473934 (128h13m38.42504225s) [upload=112562]: Waiting for error group managing segments... Dec 10 23:42:50.740943385 (128h13m38.438511691s) [noise=true , address=45.91.116.9:4291 , piece_id=XKBT7V2KFCUJXSKCKNM4PT3FF7EUDEKA3IZWARZQ4UOT2G2TWQWA, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:50.746247152 (128h13m38.443815468s) [noise=true , address=45.91.116.11:4134 , piece_id=6M24N33DZRSOQS6ECCW2N3RYRUUPGJKQEPJEAGCVZDFXC2GGD7QA, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:50.753025113 (128h13m38.450593429s) [noise=true , address=45.91.116.5:4331 , piece_id=75CMZZD7IAVEH672OX2JSB5QJRTDIXYZIPN6SMY3YDTX2J3IU4JQ, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:50.755642374 (128h13m38.453210679s) [noise=true , address=45.91.116.14:4198 , piece_id=2SR76LLNBZN4ZSBMNCVGYR6N4BAM6PG5NK2MCR3YIJ2GBG235K5Q, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:50.757756323 (128h13m38.455324639s) [noise=true , address=45.91.116.10:4114 , piece_id=4Q2LXQQWV66YRI5LVHALF2OFTV7OPFHQVEJRQTFC3DFQSHFXDABQ, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:50.771970695 (128h13m38.469539011s) [noise=true , address=145.79.227.47:4296 , piece_id=KKDVH3AYSCCOCPRAHJRGBW4HANQWGZTGDISA3F3KOFFWMR7FLJJQ, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:50.849762059 (128h13m38.547330366s) [noise=true , address=5.231.73.4:4034 , piece_id=KMX6JS77IS6EBJLPD326LJYNUKC4XTJOMQ24KTE65F42WMXSTVXA, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:50.858162923 (128h13m38.555731228s) [noise=true , address=5.231.73.5:4034 , piece_id=2GQADOR6KAUTJULRHT2MOF43ZMQCS6FG5IKWOV7PAI4OGWBEB26Q, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:50.866082536 (128h13m38.563650843s) [noise=true , address=5.231.73.4:4031 , piece_id=KNIZEUUIPZGMWEEMKZGPVH3NEIOSJMTBFOA6GAFFIEPEHSCFWR6A, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:42:51.267130776 (128h13m38.964699092s) [noise=true , address=64.49.13.138:4488 , piece_id=B32VA6WJFMYZTZ22Z565MKAMXZMUHOECMJF2NOWHCV6RNDOYDFJA, seg_pos={0 0}, upload=112562]: Done uploading piece. err: <nil> Dec 10 23:43:50.625015553 (128h14m38.322583899s) [noise=true , address=77.90.31.21:4897 , piece_id=ZQ6S4ZEVZMS4MI6YTLMDELPM64MUETF5O543FXQXIBHGMG72ZPWA, seg_pos={0 0}, upload=112562]: Done uploading piece. err: ecclient: upload failed (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, address:77.90.31.21:4897): protocol: rpc: dial tcp 77.90.31.21:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:50.625205528 (128h14m38.322773863s) [noise=true , address=77.90.31.16:4762 , piece_id=HIILELXRPQJUJICKFSPZA2CDAPLEJUFPAJATOGJ6Y43T3ENYOTKQ, seg_pos={0 0}, upload=112562]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:50.625243979 (128h14m38.322812316s) [noise=true , address=77.90.31.10:4546 , piece_id=WWWJSUNAIW7DYCLQF4DLXAT6LSJ43WRTHN4L2D2VGSVW7YVWITAQ, seg_pos={0 0}, upload=112562]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: rpc: dial tcp 77.90.31.10:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:50.625424688 (128h14m38.322993023s) [noise=true , address=77.90.31.13:4648 , piece_id=BEXDN23DVUPXHFUMYZVSH43WSWVLGAUH6WMWHUL5DBUSCMVMYO6Q, seg_pos={0 0}, upload=112562]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: rpc: dial tcp 77.90.31.13:4648: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:50.625443902 (128h14m38.323012249s) [noise=true , address=77.90.31.12:4601 , piece_id=SLIGIX7PLI4YW6PEPAWAAOO2X62N2V2WSELY4HZOVEL22YVSIG5A, seg_pos={0 0}, upload=112562]: Done uploading piece. err: ecclient: upload failed (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, address:77.90.31.12:4601): protocol: rpc: dial tcp 77.90.31.12:4601: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:50.625616795 (128h14m38.323185132s) [noise=true , address=77.90.31.11:4586 , piece_id=QWD3OSPAGIWXGOGGNZ52M73FIN4A5ELZ6OAMNTTELDBRK6UAXOKQ, seg_pos={0 0}, upload=112562]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 5.161.89.252:22874->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:50.625669935 (128h14m38.323238272s) [noise=true , address=77.90.31.15:4722 , piece_id=F2VKK5S4SSHFWTTG53ZM3SFJNYG7ATWW6X52F7NAEWJR7W2AL4SA, seg_pos={0 0}, upload=112562]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:50.625834673 (128h14m38.32340301s) [noise=true , address=77.90.31.14:4693 , piece_id=K65PGTJX7GKDCAGFDTBRLV3Z4KIZYVXOKVSYGUXQTQY2ABU4VSOA, seg_pos={0 0}, upload=112562]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: write tcp 5.161.89.252:63290->77.90.31.14:4693: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:50.625861774 (128h14m38.32343011s) [seg_pos={0 0}, upload=112562]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:43:50.640796411 (128h14m38.338364747s) [upload=112562]: Done uploading segments... Dec 10 23:43:50.640819084 (128h14m38.33838742s) [upload=112562]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.017733468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X..X.X....X.X.............] duration: 355.586158ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:43:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 355.611395ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.017703708s trace: https://jaeger.dev.storj.io/trace/64ce9b418111a1dc FAIL (2025-12-10 23:43:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 10 23:43:18.189921980 (128h14m5.887490316s) [upload=112569]: Starting upload Dec 10 23:43:18.190018921 (128h14m5.887587256s) [upload=112569]: Done starting upload Dec 10 23:43:18.190095975 (128h14m5.887664291s) [upload=112569]: Uploading segments... Dec 10 23:43:18.190466828 (128h14m5.888035173s) [upload=112569]: Got next segment. Inline: false Dec 10 23:43:18.190636645 (128h14m5.888204981s) [upload=112569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:43:18.190648617 (128h14m5.888216964s) [upload=112569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:43:18.212055755 (128h14m5.909624091s) [seg_pos={0 0}, upload=112569]: Begin upload segment... Dec 10 23:43:18.212127169 (128h14m5.909695505s) [seg_pos={0 0}, upload=112569]: Done begin upload segment. Dec 10 23:43:18.212166903 (128h14m5.90973521s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212326711 (128h14m5.909895058s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.212964534 (128h14m5.910532869s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.213649503 (128h14m5.911217879s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.214350843 (128h14m5.911919189s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.215053896 (128h14m5.912622232s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.216528440 (128h14m5.914096756s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.217484036 (128h14m5.915052372s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.219734709 (128h14m5.917303056s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.221752038 (128h14m5.919320354s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.222843087 (128h14m5.920411403s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.223950536 (128h14m5.921518871s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.224290610 (128h14m5.921858926s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.225358526 (128h14m5.922926842s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.226441568 (128h14m5.924009884s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.227761984 (128h14m5.9253303s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Uploading piece... Dec 10 23:43:18.307476872 (128h14m6.005045189s) [upload=112569]: Next returned nil segment Dec 10 23:43:18.307485960 (128h14m6.005054266s) [upload=112569]: Waiting for error group managing segments... Dec 10 23:43:18.317780600 (128h14m6.015348906s) [noise=true , address=145.79.227.38:4978 , piece_id=EPODWC2VJEJBH7UIHAMV5WFV7OIQOBNICSDNDOPVBT3K65TG7EXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.318585193 (128h14m6.016153529s) [noise=true , address=145.79.227.41:4085 , piece_id=S54DEP3752KSVXFTG6FO7MW22LRDJLRGBOQS2QARHUP7GMJYFW5A, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319271344 (128h14m6.016839661s) [noise=true , address=145.79.227.39:4013 , piece_id=2RISDAVK36POJPTJDB6YO5IDCUWEHVZMYNAYY2XFLYGOKUBEBDGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.319655953 (128h14m6.017224259s) [noise=true , address=145.79.227.43:4158 , piece_id=4ZDBJ265DBM3A54YE3IYVUIYBNC5RCYRFH45WFCTWJEH4CHJ3QPA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320043057 (128h14m6.017611373s) [noise=true , address=145.79.227.48:4302 , piece_id=ABFXZAYJN3DCSFUYL2JRN6KM7CPPUH7PX3YD7L6UCM3YO7BA2EGQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.320465927 (128h14m6.018034233s) [noise=true , address=145.79.227.45:4231 , piece_id=TWOXZIJ4RHGM2PI2ZSF2MZHX5I5Q6IXNOUYN7M6W57T7OENV5XVA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321038206 (128h14m6.018606522s) [noise=true , address=145.79.227.51:4409 , piece_id=TSZUUEW5HX526GDC5G6RSRS67WTIZDTO7VLE5OJU5Y7AWYKF3BEQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321434095 (128h14m6.019002411s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRIZL6VUJLCTML5FVRM4HK26TEHRRNEODICP6IBE72DJ5KH5FNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321500078 (128h14m6.019068414s) [noise=true , address=145.79.227.50:4375 , piece_id=564JVBUPE4HJI5B4ECLO5IIEHUN4JGRLBT5G4Y65B7VVQCO4BYLA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.321799648 (128h14m6.019367954s) [noise=true , address=145.79.227.37:4567 , piece_id=AUNBFO5O5OUHYWPM3TLO7EAC2DZLN6VRKFESYQ32L76NWFXWPYXQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.322291146 (128h14m6.019859492s) [noise=true , address=145.79.227.32:4834 , piece_id=VYSCJCBXM5GPYAADUPKNRAGWUZ7HHTUVK7HMBWWWI7CUKLJB7GJQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:18.323937782 (128h14m6.021506098s) [noise=true , address=145.79.227.28:4725 , piece_id=EGYLMQZDUV7T5ARJHSC5AOWZTPF3EJLN4YVNE7QAOEPGEHJI5PBQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: <nil> Dec 10 23:43:48.176980946 (128h14m35.874549271s) [noise=true , address=77.90.31.13:4628 , piece_id=YRBKBVLNEVVMAFVM5QRSYKRQ33KEB3UMHFBKRDC62QSHY5PCLSDA, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 5.161.89.252:20454->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177015180 (128h14m35.874583505s) [noise=true , address=77.90.31.11:4554 , piece_id=NIXH5WYIGZZABRKSH2EVI2B7BLUNORJ6O3QTSVCIYRYFMNFSNBLQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177147186 (128h14m35.874715512s) [noise=true , address=77.90.31.16:4735 , piece_id=M7UD3HA357RCQPWSXUAVWXS3BEX3YMYXA53Q5FOREUBY4IE2BJNQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:48708->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177168546 (128h14m35.874736872s) [noise=true , address=77.90.31.17:4772 , piece_id=MTVVZELXEUFARFOWBHS2Y5KG66GBLUBG5LCKNF72HSOB5NFYYCSQ, seg_pos={0 0}, upload=112569]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:43:48.177193022 (128h14m35.874761348s) [seg_pos={0 0}, upload=112569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 10 23:43:48.192954684 (128h14m35.89052299s) [upload=112569]: Done uploading segments... Dec 10 23:43:48.192970223 (128h14m35.890538539s) [upload=112569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.017721851s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................XXX.........................] duration: 610.900849ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:43:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 610.921829ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................XXX........................] duration: 560.136797ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:43:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 560.153659ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............................XX.XX........................] duration: 465.792234ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:43:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.808233ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.01507093s trace: https://jaeger.dev.storj.io/trace/39a41dfea8e14186 FAIL (2025-12-10 23:42:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tItem_ObjectBegin Dec 10 23:41:50.568803891 (128h12m38.266372207s) [upload=112549]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:41:50.590385745 (128h12m38.287954051s) [seg_pos={0 0}, upload=112549]: Begin upload segment... Dec 10 23:41:50.590427533 (128h12m38.287995839s) [seg_pos={0 0}, upload=112549]: Done begin upload segment. Dec 10 23:41:50.590459473 (128h12m38.288027789s) [noise=true , address=145.79.227.29:4948 , piece_id=GK37UTM3HRRYBBCKBC6SHWB5S7X6WP3PSPRZB63KIO7MQHB6OHQA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.590468830 (128h12m38.288037146s) [noise=true , address=77.90.31.11:4575 , piece_id=LG2LVFOGW66AZSQATTXNXAKLKIGXGXSL4FJH2GOI2DND74DPBMIA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.590872875 (128h12m38.288441191s) [noise=true , address=145.79.227.40:4059 , piece_id=VPDFIAHUR3JYG5A3YUE23L4RYO5MQ7TEGW3WRSDDTRK3L4JBTQSQ, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.590969495 (128h12m38.288537801s) [noise=true , address=5.231.73.2:4017 , piece_id=SBR6PCWFMZM3AP7GGZ3ZQ7XHPWA2XIJP75YXYAQCK6FI62B7Y4PQ, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.591219923 (128h12m38.288788239s) [noise=true , address=145.79.227.38:4984 , piece_id=QQN6DFX2TOCC3PGBI45CKN4FQ7QV4IMLJSACZTCM7J5TF3NRU7KA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.591228539 (128h12m38.288796845s) [noise=true , address=77.90.31.21:4886 , piece_id=M3AFTLOCKQ2XOPJOK4KUF4BEGI443SDMMNJGFSFCC5SLE36PSKGQ, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.591574045 (128h12m38.289142351s) [noise=true , address=45.91.116.14:4200 , piece_id=ZUGFHD66O56DKR7CGUW7N7TKFV6MSPQJ72C6YRIFYGYRVL6BINOA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.591589564 (128h12m38.28915788s) [noise=true , address=145.79.227.45:4250 , piece_id=UPBBISVTS2KGGD2TF7DCSPNSBIAWKH66X5WOSQRXE4WPSMRZ66ZA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.591857584 (128h12m38.2894259s) [noise=true , address=145.79.227.44:4223 , piece_id=MKTNFC36S6OHIBKUSMMCSWAE6ZZMYRFX4MSG2R25Z6B7YBVK6IWA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.591963482 (128h12m38.289531798s) [noise=true , address=145.79.227.34:4916 , piece_id=23UABWTSAYS74BIFSDEPPSCEQNCQQNSIPPVQPKMNR3JSCYG6ZF7A, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.592147196 (128h12m38.289715512s) [noise=true , address=77.90.31.12:4604 , piece_id=KPK6ZID64L7Q2XE5XU5QNZAPTAD5SNRDIJVZEDQOS7XFEH4JPR3Q, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.592352077 (128h12m38.289920424s) [noise=true , address=77.90.31.15:4724 , piece_id=BB7LJFUSRLCY3HMYQPHD3LWHEN56HZLDRCFYTCPKJYZIAQCBVCNQ, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.592422850 (128h12m38.289991166s) [noise=true , address=77.90.31.13:4630 , piece_id=KJQIK5N7HDS2APDFPZNP52AW6JVPRAYKVJT5CVWBCEU5S6BQETSA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.592724585 (128h12m38.29029289s) [noise=true , address=145.79.227.9:4394 , piece_id=Q5M4HKO2VFFUICVKJXFELVL2KVWUAVE6YWUB4A22LXIUSZYE43KA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.592853765 (128h12m38.290422072s) [noise=true , address=45.91.116.6:4977 , piece_id=TZ4EWB736OUAKFPUVUDTKF2PEP2IL23MS4AKLSWEAB5EJ23TYXLQ, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.593063998 (128h12m38.290632304s) [noise=true , address=45.91.116.10:4098 , piece_id=G4NWFRHLBJNH6XE2KJILBYWQ62BLO7C6WEQNQXZGJAQ43KULOW5A, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.593312261 (128h12m38.290880577s) [noise=true , address=77.90.31.18:4834 , piece_id=N2JNFGASHRC2Q2RMCGVAVM4VHH4NI2TDZWNWIMQDY23BWXV5SIRA, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.593347848 (128h12m38.290916164s) [noise=true , address=64.49.13.26:4949 , piece_id=BOIECSUNJ573O4QBTUNZNXLPFN6JEVE5VQL24EM4VNHEO2HFEJ3A, seg_pos={0 0}, upload=112549]: Uploading piece... Dec 10 23:41:50.665215996 (128h12m38.362784302s) [upload=112549]: Next returned nil segment Dec 10 23:41:50.665225003 (128h12m38.362793309s) [upload=112549]: Waiting for error group managing segments... Dec 10 23:41:50.680254096 (128h12m38.377822412s) [noise=true , address=145.79.227.9:4394 , piece_id=Q5M4HKO2VFFUICVKJXFELVL2KVWUAVE6YWUB4A22LXIUSZYE43KA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:50.682319285 (128h12m38.379887591s) [noise=true , address=45.91.116.6:4977 , piece_id=TZ4EWB736OUAKFPUVUDTKF2PEP2IL23MS4AKLSWEAB5EJ23TYXLQ, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:50.685258405 (128h12m38.382826721s) [noise=true , address=45.91.116.14:4200 , piece_id=ZUGFHD66O56DKR7CGUW7N7TKFV6MSPQJ72C6YRIFYGYRVL6BINOA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:50.690709668 (128h12m38.388277984s) [noise=true , address=45.91.116.10:4098 , piece_id=G4NWFRHLBJNH6XE2KJILBYWQ62BLO7C6WEQNQXZGJAQ43KULOW5A, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:50.693723969 (128h12m38.391292285s) [noise=true , address=145.79.227.29:4948 , piece_id=GK37UTM3HRRYBBCKBC6SHWB5S7X6WP3PSPRZB63KIO7MQHB6OHQA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:50.719407083 (128h12m38.4169754s) [noise=true , address=145.79.227.34:4916 , piece_id=23UABWTSAYS74BIFSDEPPSCEQNCQQNSIPPVQPKMNR3JSCYG6ZF7A, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:50.738014662 (128h12m38.435582968s) [noise=true , address=145.79.227.40:4059 , piece_id=VPDFIAHUR3JYG5A3YUE23L4RYO5MQ7TEGW3WRSDDTRK3L4JBTQSQ, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:50.792460374 (128h12m38.49002869s) [noise=true , address=5.231.73.2:4017 , piece_id=SBR6PCWFMZM3AP7GGZ3ZQ7XHPWA2XIJP75YXYAQCK6FI62B7Y4PQ, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:51.181040270 (128h12m38.878608576s) [noise=true , address=64.49.13.26:4949 , piece_id=BOIECSUNJ573O4QBTUNZNXLPFN6JEVE5VQL24EM4VNHEO2HFEJ3A, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:51.301648995 (128h12m38.9992173s) [noise=true , address=145.79.227.45:4250 , piece_id=UPBBISVTS2KGGD2TF7DCSPNSBIAWKH66X5WOSQRXE4WPSMRZ66ZA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:51.558419434 (128h12m39.255987769s) [noise=true , address=145.79.227.44:4223 , piece_id=MKTNFC36S6OHIBKUSMMCSWAE6ZZMYRFX4MSG2R25Z6B7YBVK6IWA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:41:51.629195291 (128h12m39.326763627s) [noise=true , address=145.79.227.38:4984 , piece_id=QQN6DFX2TOCC3PGBI45CKN4FQ7QV4IMLJSACZTCM7J5TF3NRU7KA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: <nil> Dec 10 23:42:50.556793522 (128h13m38.254361868s) [noise=true , address=77.90.31.11:4575 , piece_id=LG2LVFOGW66AZSQATTXNXAKLKIGXGXSL4FJH2GOI2DND74DPBMIA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 5.161.89.252:17710->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:50.556991462 (128h13m38.254559808s) [noise=true , address=77.90.31.13:4630 , piece_id=KJQIK5N7HDS2APDFPZNP52AW6JVPRAYKVJT5CVWBCEU5S6BQETSA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:50.557009776 (128h13m38.254578123s) [noise=true , address=77.90.31.21:4886 , piece_id=M3AFTLOCKQ2XOPJOK4KUF4BEGI443SDMMNJGFSFCC5SLE36PSKGQ, seg_pos={0 0}, upload=112549]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: rpc: dial tcp 77.90.31.21:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:50.557218317 (128h13m38.254786652s) [noise=true , address=77.90.31.12:4604 , piece_id=KPK6ZID64L7Q2XE5XU5QNZAPTAD5SNRDIJVZEDQOS7XFEH4JPR3Q, seg_pos={0 0}, upload=112549]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:50.557304657 (128h13m38.254873004s) [noise=true , address=77.90.31.15:4724 , piece_id=BB7LJFUSRLCY3HMYQPHD3LWHEN56HZLDRCFYTCPKJYZIAQCBVCNQ, seg_pos={0 0}, upload=112549]: Done uploading piece. err: ecclient: upload failed (node:12YEcohZBsQSm45orPuT8np4EGgXQaWAM8rCsmUfA1PfNubwLmo, address:77.90.31.15:4724): protocol: rpc: dial tcp 77.90.31.15:4724: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:50.557538984 (128h13m38.255107321s) [noise=true , address=77.90.31.18:4834 , piece_id=N2JNFGASHRC2Q2RMCGVAVM4VHH4NI2TDZWNWIMQDY23BWXV5SIRA, seg_pos={0 0}, upload=112549]: Done uploading piece. err: ecclient: upload failed (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q, address:77.90.31.18:4834): protocol: rpc: dial tcp 77.90.31.18:4834: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:42:50.557569292 (128h13m38.255137628s) [seg_pos={0 0}, upload=112549]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:42:50.570365365 (128h13m38.2679337s) [upload=112549]: Done uploading segments... Dec 10 23:42:50.570389200 (128h13m38.267957545s) [upload=112549]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.015082933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................X..X.X....X.X............] duration: 342.724722ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:42:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 342.741123ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................X..X.X....X.X...........] duration: 346.343783ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:42:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 346.355745ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................................XX.XX.......................] duration: 485.657041ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:42:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.67242ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration: 551.388929ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:42:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 551.406542ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................................XX.XX......................] duration: 455.191231ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:41:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 455.208793ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X..X.X....X.X..........] duration: 373.119797ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 373.133714ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m0.019891826s trace: https://jaeger.dev.storj.io/trace/201931f6ae849bb8 FAIL (2025-12-10 23:41:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== :50.508022033 (128h11m38.205590339s) [upload=112534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:40:50.508028325 (128h11m38.20559663s) [upload=112534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:40:50.530713751 (128h11m38.228282057s) [seg_pos={0 0}, upload=112534]: Begin upload segment... Dec 10 23:40:50.530747454 (128h11m38.22831576s) [seg_pos={0 0}, upload=112534]: Done begin upload segment. Dec 10 23:40:50.530782869 (128h11m38.228351185s) [noise=true , address=77.90.31.10:4546 , piece_id=3LHNG5GPGCKOQI7YNSBZZXDGX72B4RDXPSVK3DUXY35MX4IHKXNQ, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.530800442 (128h11m38.228368748s) [noise=true , address=77.90.31.19:4849 , piece_id=PCIST2WAXD7BIRTYA5VMI4IUCY7SXWG7KYNKT6TGBQTPYJXGNC5A, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.531227970 (128h11m38.228796277s) [noise=true , address=45.91.116.4:4948 , piece_id=GOAJW6K6H25F4DAAZN2SOYM37OODM4LMVE2CO4DTFCGITO726MAQ, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.531580600 (128h11m38.229148906s) [noise=true , address=45.91.116.6:4006 , piece_id=LVI2OVT6LNLPUQZI5WQ3RM6XD6J6Z7TYGTHNHGH5OCQS2O3RZM3Q, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.531957684 (128h11m38.22952601s) [noise=true , address=77.90.31.18:4816 , piece_id=LTBOYMA7Y67L6YU3FLSWMZCE2CEUA63RUKHMKFZGGEG7S7SMBWDA, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.532310093 (128h11m38.229878409s) [noise=true , address=45.91.116.12:4246 , piece_id=OD7TAL7SV23ZCY3JVO4W67IB6R75XORDFMWNY3QURKJGCUEPMVZA, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.532700311 (128h11m38.230268618s) [noise=true , address=145.79.227.29:4955 , piece_id=GUUJL3I6N7T3T7KLPWMRZMQAXCD3DRF52I3LCJ2IB7DXJWAH6VLA, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.533034425 (128h11m38.230602731s) [noise=true , address=77.90.31.12:4611 , piece_id=S4VGT6B2MG3KPC5H3GQQJ4PFNP6R6ZNFEOGPAJIFXM3QNEZ7CLYA, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.533451013 (128h11m38.231019319s) [noise=true , address=145.79.227.13:4831 , piece_id=RJLDYXDNTEN65S4KVMR3ALZZGCCUVQ3IMDV3WLVEHLD74HNOAZFA, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.535159394 (128h11m38.2327277s) [noise=true , address=145.79.227.45:4245 , piece_id=3ZW33UWSVU2LG3SNS5263CNINQCQZA6TC6ILMSU7L6BOUA4NKV6A, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.536404531 (128h11m38.233972836s) [noise=true , address=45.91.116.11:4135 , piece_id=6WAGYQNWZHB5JEJLAFBVMTRYP7VG5G7XHBWN55CGYTOFBNDOTVOQ, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.537577502 (128h11m38.235145808s) [noise=true , address=77.90.31.17:4793 , piece_id=7N5QY555BMQK7AT65PR2M2LELWA63MPRQQRNFCRMINJZDZEWSDUA, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.538810465 (128h11m38.23637877s) [noise=true , address=45.91.116.14:4209 , piece_id=F3DLB3FQKVKL2GILB2YRDKCZOABU2P3SWLJ5CRG3XLOVVIGAUAWQ, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.540480114 (128h11m38.238048439s) [noise=true , address=64.49.13.170:4722 , piece_id=GYU2K7RMXA2SCC6RWH4FZBBLLSJETXABO3ENRWPURA6MISAQTW6A, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.542427992 (128h11m38.239996318s) [noise=true , address=77.90.31.16:4766 , piece_id=C6WGHGM6KSNTMWII47SWRAAGFOGZKTT4ZZ6VRQ77BBXT566PG4UQ, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.543004889 (128h11m38.240573205s) [noise=true , address=5.231.73.3:4032 , piece_id=JRQBZWZT53FRUESSCJ2WUPUU3G6CGDQSQ6MRCQW5QZ3ICDWSDVPQ, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.544112308 (128h11m38.241680624s) [noise=true , address=45.91.116.11:4142 , piece_id=4CZPQ4IMKEVB4MJMEQVLX6IW4SNEV7EMG44XQH5QKHS7RHIY3CPA, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.545765326 (128h11m38.243333642s) [noise=true , address=64.49.13.154:4925 , piece_id=FU3AA56D5ZPZED5YDWKMZDPHOVKMM34B5GW73CBE5ZYJNZKGV75A, seg_pos={0 0}, upload=112534]: Uploading piece... Dec 10 23:40:50.595311056 (128h11m38.292879362s) [upload=112534]: Next returned nil segment Dec 10 23:40:50.595338668 (128h11m38.292906974s) [upload=112534]: Waiting for error group managing segments... Dec 10 23:40:50.604005196 (128h11m38.301573512s) [noise=true , address=45.91.116.4:4948 , piece_id=GOAJW6K6H25F4DAAZN2SOYM37OODM4LMVE2CO4DTFCGITO726MAQ, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.609190843 (128h11m38.306759149s) [noise=true , address=45.91.116.11:4135 , piece_id=6WAGYQNWZHB5JEJLAFBVMTRYP7VG5G7XHBWN55CGYTOFBNDOTVOQ, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.612331971 (128h11m38.309900287s) [noise=true , address=45.91.116.6:4006 , piece_id=LVI2OVT6LNLPUQZI5WQ3RM6XD6J6Z7TYGTHNHGH5OCQS2O3RZM3Q, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.619418478 (128h11m38.316986784s) [noise=true , address=45.91.116.11:4142 , piece_id=4CZPQ4IMKEVB4MJMEQVLX6IW4SNEV7EMG44XQH5QKHS7RHIY3CPA, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.619656573 (128h11m38.317224889s) [noise=true , address=145.79.227.45:4245 , piece_id=3ZW33UWSVU2LG3SNS5263CNINQCQZA6TC6ILMSU7L6BOUA4NKV6A, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.620430579 (128h11m38.317998885s) [noise=true , address=45.91.116.12:4246 , piece_id=OD7TAL7SV23ZCY3JVO4W67IB6R75XORDFMWNY3QURKJGCUEPMVZA, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.622899430 (128h11m38.320467737s) [noise=true , address=45.91.116.14:4209 , piece_id=F3DLB3FQKVKL2GILB2YRDKCZOABU2P3SWLJ5CRG3XLOVVIGAUAWQ, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.625177055 (128h11m38.322745371s) [noise=true , address=145.79.227.13:4831 , piece_id=RJLDYXDNTEN65S4KVMR3ALZZGCCUVQ3IMDV3WLVEHLD74HNOAZFA, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.735007205 (128h11m38.432575512s) [noise=true , address=5.231.73.3:4032 , piece_id=JRQBZWZT53FRUESSCJ2WUPUU3G6CGDQSQ6MRCQW5QZ3ICDWSDVPQ, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:50.812130920 (128h11m38.509699226s) [noise=true , address=145.79.227.29:4955 , piece_id=GUUJL3I6N7T3T7KLPWMRZMQAXCD3DRF52I3LCJ2IB7DXJWAH6VLA, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:51.102836189 (128h11m38.800404495s) [noise=true , address=64.49.13.154:4925 , piece_id=FU3AA56D5ZPZED5YDWKMZDPHOVKMM34B5GW73CBE5ZYJNZKGV75A, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:40:51.142864912 (128h11m38.840433219s) [noise=true , address=64.49.13.170:4722 , piece_id=GYU2K7RMXA2SCC6RWH4FZBBLLSJETXABO3ENRWPURA6MISAQTW6A, seg_pos={0 0}, upload=112534]: Done uploading piece. err: <nil> Dec 10 23:41:50.495081409 (128h12m38.192649725s) [noise=true , address=77.90.31.12:4611 , piece_id=S4VGT6B2MG3KPC5H3GQQJ4PFNP6R6ZNFEOGPAJIFXM3QNEZ7CLYA, seg_pos={0 0}, upload=112534]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: rpc: dial tcp 77.90.31.12:4611: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:50.495101747 (128h12m38.192670064s) [noise=true , address=77.90.31.19:4849 , piece_id=PCIST2WAXD7BIRTYA5VMI4IUCY7SXWG7KYNKT6TGBQTPYJXGNC5A, seg_pos={0 0}, upload=112534]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: rpc: dial tcp 77.90.31.19:4849: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:50.495226490 (128h12m38.192794806s) [noise=true , address=77.90.31.16:4766 , piece_id=C6WGHGM6KSNTMWII47SWRAAGFOGZKTT4ZZ6VRQ77BBXT566PG4UQ, seg_pos={0 0}, upload=112534]: Done uploading piece. err: ecclient: upload failed (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK, address:77.90.31.16:4766): protocol: rpc: dial tcp 77.90.31.16:4766: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:50.495343709 (128h12m38.192912015s) [noise=true , address=77.90.31.10:4546 , piece_id=3LHNG5GPGCKOQI7YNSBZZXDGX72B4RDXPSVK3DUXY35MX4IHKXNQ, seg_pos={0 0}, upload=112534]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: rpc: dial tcp 77.90.31.10:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:50.495588346 (128h12m38.193156662s) [noise=true , address=77.90.31.18:4816 , piece_id=LTBOYMA7Y67L6YU3FLSWMZCE2CEUA63RUKHMKFZGGEG7S7SMBWDA, seg_pos={0 0}, upload=112534]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: write tcp 5.161.89.252:64898->77.90.31.18:4816: use of closed network connection; context deadline exceeded Dec 10 23:41:50.495674307 (128h12m38.193242623s) [noise=true , address=77.90.31.17:4793 , piece_id=7N5QY555BMQK7AT65PR2M2LELWA63MPRQQRNFCRMINJZDZEWSDUA, seg_pos={0 0}, upload=112534]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:41:50.495694585 (128h12m38.193262901s) [seg_pos={0 0}, upload=112534]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:41:50.513876638 (128h12m38.211444954s) [upload=112534]: Done uploading segments... Dec 10 23:41:50.513896225 (128h12m38.211464541s) [upload=112534]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.019900832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................XXX......................] duration: 621.976225ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:41:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 622.005771ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration: 576.784865ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:41:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 576.806985ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................................XX.XX.....................] duration: 538.201865ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:40:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.216924ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................X..X.X....X.X.........] duration: 313.582622ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:40:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 313.59737ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 1m0.019578145s trace: https://jaeger.dev.storj.io/trace/51e9c0fa676780a9 FAIL (2025-12-10 23:40:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 44756009s) [upload=112519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:39:50.469060851 (128h10m38.166629167s) [seg_pos={0 0}, upload=112519]: Begin upload segment... Dec 10 23:39:50.469083633 (128h10m38.166651939s) [seg_pos={0 0}, upload=112519]: Done begin upload segment. Dec 10 23:39:50.469107769 (128h10m38.166676074s) [noise=true , address=77.90.31.21:4899 , piece_id=HNXWHUXBMKEGXS6OXHWKH7QQCTNJ3QQIGWG7P6PFYLAZFGXLWF7A, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.469115824 (128h10m38.16668414s) [noise=true , address=77.90.31.14:4688 , piece_id=PAQJVT44RMNMDP3NP365A3JN7CI3YRNK3VGJJIVBRLSUACOEX4XQ, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.469404032 (128h10m38.166972348s) [noise=true , address=77.90.31.12:4620 , piece_id=L7OSLNHBGRUTYXZIWNWNVAZEYV37GACQXWUCUOVFHMZJJUYALE7Q, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.469695647 (128h10m38.167263953s) [noise=true , address=145.79.227.43:4185 , piece_id=OVI5OOWFXDHI6ROFXGQKRLZTJB7VCDWHYMSJP4IMKFWOZUB7BFWA, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.469936216 (128h10m38.167504532s) [noise=true , address=45.91.116.11:4129 , piece_id=WI5FVIXLWFAZB6LK7LFOB32HPY5TYLVCSTEO6E5IAGFJHRUCERXQ, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.470167348 (128h10m38.167735664s) [noise=true , address=77.90.31.17:4795 , piece_id=DWALV267ZZGUE3M5Q4WMNJTSEW5JBLAWC6TLRXRQDJSD2H7WJKRA, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.470401666 (128h10m38.167969982s) [noise=true , address=45.91.116.10:4099 , piece_id=MZGZCELCOLVNECD273VJX7ZGJOU2Z4PP5CYNJG3SUMD4UDM4BTDA, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.470652003 (128h10m38.168220309s) [noise=true , address=5.231.73.4:4034 , piece_id=GWG6YZBYQCMTGWKTR3E3JLYJPIRHNU2E6ICIVXALX2SLJA23EKFA, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.470655400 (128h10m38.168223716s) [noise=true , address=77.90.31.16:4746 , piece_id=7C5FD2D5GGDTRX2AISMSYWRSGFNS5D7QXDEPC7CKCMOTB34RBVJQ, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.470907611 (128h10m38.168475927s) [noise=true , address=145.79.227.50:4386 , piece_id=6AIHXO2IEUQXXQZFKWXKYFK7JDPMXC7SCLNNEMQ4GKRKJFMUMNNQ, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.470913462 (128h10m38.168481767s) [noise=true , address=45.91.116.6:4986 , piece_id=SAC6WPMGT5KKDYUWCNNC4LDXS2HME2BKHSQFDE5SMJB7IPJQEZ3Q, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.471149312 (128h10m38.168717618s) [noise=true , address=145.79.227.34:4929 , piece_id=P42LCPDKLBVYY25I7LH4MZNFS4KCR2L243J4V57CTX6XQHABEW4Q, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.471175771 (128h10m38.168744078s) [noise=true , address=145.79.227.41:4095 , piece_id=4N6LGQVU3BBA5CPBLZ4X5U7I3W3JPJU6CMY37YMAR6XNEAWW4ASA, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.471426349 (128h10m38.168994666s) [noise=true , address=145.79.227.11:4733 , piece_id=JVGI4WWQM6F2ORB7HUYDBFTFZNBPDKHVCEFUXGUGUQVO4OVCHOIA, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.471714217 (128h10m38.169282533s) [noise=true , address=45.91.116.9:4270 , piece_id=LE62VJDIYNEEWXOZXLTGRBVKVBL3WF7UA7AIMYJWGVN3DBTEUDYA, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.472017494 (128h10m38.16958581s) [noise=true , address=77.90.31.18:4819 , piece_id=FEXQ4KCHIPBHQXJDPDLDVDNPRDDLH5HXWLS732MIXM7GZOJ67ARQ, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.472254917 (128h10m38.169823233s) [noise=true , address=145.79.227.44:4217 , piece_id=ZAXFQAMUMBAKHS2XI4WXSUEY2SQ2HEVPMHU7253LSO74AEXUF2BQ, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.472550800 (128h10m38.170119116s) [noise=true , address=64.49.13.27:4961 , piece_id=XKZXY5NXS5EKTF6FAB6J5N2U3WZORBBWOV6LVKLFJMT5YDWVHIRA, seg_pos={0 0}, upload=112519]: Uploading piece... Dec 10 23:39:50.565372147 (128h10m38.262940483s) [upload=112519]: Next returned nil segment Dec 10 23:39:50.565480510 (128h10m38.263048846s) [upload=112519]: Waiting for error group managing segments... Dec 10 23:39:50.574360396 (128h10m38.271928722s) [noise=true , address=145.79.227.50:4386 , piece_id=6AIHXO2IEUQXXQZFKWXKYFK7JDPMXC7SCLNNEMQ4GKRKJFMUMNNQ, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:50.576415435 (128h10m38.273983761s) [noise=true , address=145.79.227.11:4733 , piece_id=JVGI4WWQM6F2ORB7HUYDBFTFZNBPDKHVCEFUXGUGUQVO4OVCHOIA, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:50.584092947 (128h10m38.281661283s) [noise=true , address=45.91.116.10:4099 , piece_id=MZGZCELCOLVNECD273VJX7ZGJOU2Z4PP5CYNJG3SUMD4UDM4BTDA, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:50.585211707 (128h10m38.282780043s) [noise=true , address=45.91.116.9:4270 , piece_id=LE62VJDIYNEEWXOZXLTGRBVKVBL3WF7UA7AIMYJWGVN3DBTEUDYA, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:50.586617593 (128h10m38.284185918s) [noise=true , address=145.79.227.34:4929 , piece_id=P42LCPDKLBVYY25I7LH4MZNFS4KCR2L243J4V57CTX6XQHABEW4Q, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:50.587175895 (128h10m38.284744231s) [noise=true , address=145.79.227.43:4185 , piece_id=OVI5OOWFXDHI6ROFXGQKRLZTJB7VCDWHYMSJP4IMKFWOZUB7BFWA, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:50.587325525 (128h10m38.284893841s) [noise=true , address=45.91.116.11:4129 , piece_id=WI5FVIXLWFAZB6LK7LFOB32HPY5TYLVCSTEO6E5IAGFJHRUCERXQ, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:50.590694859 (128h10m38.288263205s) [noise=true , address=45.91.116.6:4986 , piece_id=SAC6WPMGT5KKDYUWCNNC4LDXS2HME2BKHSQFDE5SMJB7IPJQEZ3Q, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:50.669545730 (128h10m38.367114066s) [noise=true , address=5.231.73.4:4034 , piece_id=GWG6YZBYQCMTGWKTR3E3JLYJPIRHNU2E6ICIVXALX2SLJA23EKFA, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:51.050308254 (128h10m38.74787659s) [noise=true , address=64.49.13.27:4961 , piece_id=XKZXY5NXS5EKTF6FAB6J5N2U3WZORBBWOV6LVKLFJMT5YDWVHIRA, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:51.128078997 (128h10m38.825647313s) [noise=true , address=145.79.227.41:4095 , piece_id=4N6LGQVU3BBA5CPBLZ4X5U7I3W3JPJU6CMY37YMAR6XNEAWW4ASA, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:39:51.553195533 (128h10m39.250763868s) [noise=true , address=145.79.227.44:4217 , piece_id=ZAXFQAMUMBAKHS2XI4WXSUEY2SQ2HEVPMHU7253LSO74AEXUF2BQ, seg_pos={0 0}, upload=112519]: Done uploading piece. err: <nil> Dec 10 23:40:50.433980065 (128h11m38.131548381s) [noise=true , address=77.90.31.14:4688 , piece_id=PAQJVT44RMNMDP3NP365A3JN7CI3YRNK3VGJJIVBRLSUACOEX4XQ, seg_pos={0 0}, upload=112519]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:50.434077026 (128h11m38.131645331s) [noise=true , address=77.90.31.16:4746 , piece_id=7C5FD2D5GGDTRX2AISMSYWRSGFNS5D7QXDEPC7CKCMOTB34RBVJQ, seg_pos={0 0}, upload=112519]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 5.161.89.252:52964->77.90.31.16:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:50.434142468 (128h11m38.131710785s) [noise=true , address=77.90.31.18:4819 , piece_id=FEXQ4KCHIPBHQXJDPDLDVDNPRDDLH5HXWLS732MIXM7GZOJ67ARQ, seg_pos={0 0}, upload=112519]: Done uploading piece. err: ecclient: upload failed (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, address:77.90.31.18:4819): protocol: rpc: dial tcp 77.90.31.18:4819: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:50.434250610 (128h11m38.131818916s) [noise=true , address=77.90.31.21:4899 , piece_id=HNXWHUXBMKEGXS6OXHWKH7QQCTNJ3QQIGWG7P6PFYLAZFGXLWF7A, seg_pos={0 0}, upload=112519]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 5.161.89.252:58740->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:50.434385532 (128h11m38.131953838s) [noise=true , address=77.90.31.12:4620 , piece_id=L7OSLNHBGRUTYXZIWNWNVAZEYV37GACQXWUCUOVFHMZJJUYALE7Q, seg_pos={0 0}, upload=112519]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:50.434753449 (128h11m38.132321755s) [noise=true , address=77.90.31.17:4795 , piece_id=DWALV267ZZGUE3M5Q4WMNJTSEW5JBLAWC6TLRXRQDJSD2H7WJKRA, seg_pos={0 0}, upload=112519]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: rpc: dial tcp 77.90.31.17:4795: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 10 23:40:50.434785068 (128h11m38.132353374s) [seg_pos={0 0}, upload=112519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:40:50.451658757 (128h11m38.149227073s) [upload=112519]: Done uploading segments... Dec 10 23:40:50.451685918 (128h11m38.149254235s) [upload=112519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.019586751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................................XX.XX....................] duration: 490.840364ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:40:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 490.859279ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration: 564.568217ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:40:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 564.590338ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 1m0.020040365s trace: https://jaeger.dev.storj.io/trace/50f361a577cdc3c9 FAIL (2025-12-10 23:39:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:38:50.390177369 (128h9m38.087745705s) [upload=112506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:38:50.412918518 (128h9m38.110486854s) [seg_pos={0 0}, upload=112506]: Begin upload segment... Dec 10 23:38:50.412972760 (128h9m38.110541106s) [seg_pos={0 0}, upload=112506]: Done begin upload segment. Dec 10 23:38:50.413018776 (128h9m38.110587112s) [noise=true , address=45.91.116.12:4233 , piece_id=AR4VMJ2LUJRQD4VXUTLO7FHLUJYNL6IEZ2S5KABW663MRDFJVI3Q, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.413031600 (128h9m38.110599916s) [noise=true , address=5.231.73.2:4031 , piece_id=2WF7DRAOLUJ47DZD6ZM4N5H3J3NQGTFFDSAVKH4KLAMQJMXHUWVA, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.413547223 (128h9m38.111115559s) [noise=true , address=45.91.116.13:4168 , piece_id=VC2ICVT3ZLH5LON73PMDJJLHDO3PK2HK5EK3IPGKFSU4MIPQPVOQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.413594882 (128h9m38.111163219s) [noise=true , address=145.79.227.16:4942 , piece_id=VLWPKZRY42FXF6HZBWB33HGJ46PCMAY6FNOXZZP7MB4CSKB3BLJA, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.414042378 (128h9m38.111610723s) [noise=true , address=77.90.31.15:4731 , piece_id=DIOHRUDGJ2Y7L4V5V4IBJSE72JT6A6PLDRVFY3WGLZJHLSF23A4Q, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.414437246 (128h9m38.112005582s) [noise=true , address=77.90.31.10:4543 , piece_id=W2APNQAX7M4PIMLG2YWFPDBDKQU56GSG736XUSEJD5FSGRH3XU7Q, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.414866426 (128h9m38.112434763s) [noise=true , address=45.91.116.14:4195 , piece_id=TTZHR74MWMWL4E3RR46764T34BAS7X7MKNB2JHBWZEHZ2Y4CHYMQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.415469944 (128h9m38.11303825s) [noise=true , address=45.91.116.9:4290 , piece_id=TBY3DILVTOFCWGBALRWKSWDS3QSY4OU4DIHVHXAZVGXC3TGXJAGQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.415780956 (128h9m38.113349262s) [noise=true , address=145.79.227.4:4219 , piece_id=O6MUIUUJKBVJAVUXWPUNSO6ZXFGV5RQ42SET3QABIAGKAXCJQZWA, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.416049998 (128h9m38.113618304s) [noise=true , address=77.90.31.11:4559 , piece_id=5SNMRYC7YP4DBMSM5GANI2NCMNV4UZNCT6QCXTRGM76BQZ2WLKRQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.416107757 (128h9m38.113676092s) [noise=true , address=77.90.31.18:4823 , piece_id=MVII6RACELENZ5VKQQRLZKAE76TIDQK543I23GBGE2SBZ6LMWHRA, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.416602019 (128h9m38.114170355s) [noise=true , address=145.79.227.39:4035 , piece_id=KZ43LZNNK6CIX3BCYPUGXFCWVIFXLZT6EGAL5BSOWM2AJSXII3QQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.417140995 (128h9m38.114709332s) [noise=true , address=145.79.227.44:4227 , piece_id=NDRFVBZIY3DLFLJXNVDDOKIPQRNPSALNT7EHVUMU6MRH6X43INMQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.417554327 (128h9m38.115122624s) [noise=true , address=77.90.31.17:4799 , piece_id=ERMZDVKXQSLFYYINQ7ZOL6FAWBERIMTPSCGTUYAQXJN3BAX3EVMQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.417846223 (128h9m38.115414529s) [noise=true , address=5.231.73.4:4022 , piece_id=EBZBVFLLJ6WLLSPD5VURONQB3SIZGIS4INAUH4YKLN5OOOEJDG2A, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.418097412 (128h9m38.115665718s) [noise=true , address=145.79.227.12:4796 , piece_id=A72Q6SQM7YKSE6XQEOHHZDPUX4TDHSB6354ZNJTVXCCQ5KDDRHHA, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.418341518 (128h9m38.115909815s) [noise=true , address=45.91.116.5:4305 , piece_id=CYIEMZIVVEF7ZTO7HKXJYAIFCJX4YD2OYXNYCQAVD2JIVQ3QCTPQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.418459098 (128h9m38.116027444s) [noise=true , address=77.90.31.21:4907 , piece_id=YBTAKXI25STG3STLY76FZCJC3MVI56V6TUPTKE7Y2ZXOLUPR4IIQ, seg_pos={0 0}, upload=112506]: Uploading piece... Dec 10 23:38:50.503459927 (128h9m38.201028232s) [upload=112506]: Next returned nil segment Dec 10 23:38:50.503467370 (128h9m38.201035677s) [upload=112506]: Waiting for error group managing segments... Dec 10 23:38:50.511489947 (128h9m38.209058253s) [noise=true , address=45.91.116.13:4168 , piece_id=VC2ICVT3ZLH5LON73PMDJJLHDO3PK2HK5EK3IPGKFSU4MIPQPVOQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.513844635 (128h9m38.211412981s) [noise=true , address=145.79.227.39:4035 , piece_id=KZ43LZNNK6CIX3BCYPUGXFCWVIFXLZT6EGAL5BSOWM2AJSXII3QQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.519345882 (128h9m38.216914218s) [noise=true , address=145.79.227.12:4796 , piece_id=A72Q6SQM7YKSE6XQEOHHZDPUX4TDHSB6354ZNJTVXCCQ5KDDRHHA, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.528704283 (128h9m38.226272599s) [noise=true , address=45.91.116.14:4195 , piece_id=TTZHR74MWMWL4E3RR46764T34BAS7X7MKNB2JHBWZEHZ2Y4CHYMQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.529834784 (128h9m38.22740309s) [noise=true , address=145.79.227.4:4219 , piece_id=O6MUIUUJKBVJAVUXWPUNSO6ZXFGV5RQ42SET3QABIAGKAXCJQZWA, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.532574924 (128h9m38.230143259s) [noise=true , address=45.91.116.5:4305 , piece_id=CYIEMZIVVEF7ZTO7HKXJYAIFCJX4YD2OYXNYCQAVD2JIVQ3QCTPQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.540830844 (128h9m38.23839916s) [noise=true , address=45.91.116.9:4290 , piece_id=TBY3DILVTOFCWGBALRWKSWDS3QSY4OU4DIHVHXAZVGXC3TGXJAGQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.600953052 (128h9m38.298521368s) [noise=true , address=5.231.73.2:4031 , piece_id=2WF7DRAOLUJ47DZD6ZM4N5H3J3NQGTFFDSAVKH4KLAMQJMXHUWVA, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.625135273 (128h9m38.322703589s) [noise=true , address=5.231.73.4:4022 , piece_id=EBZBVFLLJ6WLLSPD5VURONQB3SIZGIS4INAUH4YKLN5OOOEJDG2A, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:50.738095743 (128h9m38.435664059s) [noise=true , address=45.91.116.12:4233 , piece_id=AR4VMJ2LUJRQD4VXUTLO7FHLUJYNL6IEZ2S5KABW663MRDFJVI3Q, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:51.357475254 (128h9m39.05504359s) [noise=true , address=145.79.227.16:4942 , piece_id=VLWPKZRY42FXF6HZBWB33HGJ46PCMAY6FNOXZZP7MB4CSKB3BLJA, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:38:51.452713405 (128h9m39.150281742s) [noise=true , address=145.79.227.44:4227 , piece_id=NDRFVBZIY3DLFLJXNVDDOKIPQRNPSALNT7EHVUMU6MRH6X43INMQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: <nil> Dec 10 23:39:50.371173210 (128h10m38.068741547s) [noise=true , address=77.90.31.17:4799 , piece_id=ERMZDVKXQSLFYYINQ7ZOL6FAWBERIMTPSCGTUYAQXJN3BAX3EVMQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:50.371386810 (128h10m38.068955146s) [noise=true , address=77.90.31.15:4731 , piece_id=DIOHRUDGJ2Y7L4V5V4IBJSE72JT6A6PLDRVFY3WGLZJHLSF23A4Q, seg_pos={0 0}, upload=112506]: Done uploading piece. err: ecclient: upload failed (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo, address:77.90.31.15:4731): protocol: rpc: dial tcp 77.90.31.15:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:50.371459785 (128h10m38.069028122s) [noise=true , address=77.90.31.10:4543 , piece_id=W2APNQAX7M4PIMLG2YWFPDBDKQU56GSG736XUSEJD5FSGRH3XU7Q, seg_pos={0 0}, upload=112506]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:50.371473842 (128h10m38.069042168s) [noise=true , address=77.90.31.11:4559 , piece_id=5SNMRYC7YP4DBMSM5GANI2NCMNV4UZNCT6QCXTRGM76BQZ2WLKRQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: rpc: dial tcp 77.90.31.11:4559: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:50.371607442 (128h10m38.069175777s) [noise=true , address=77.90.31.21:4907 , piece_id=YBTAKXI25STG3STLY76FZCJC3MVI56V6TUPTKE7Y2ZXOLUPR4IIQ, seg_pos={0 0}, upload=112506]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: rpc: dial tcp 77.90.31.21:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:50.371653708 (128h10m38.069222045s) [noise=true , address=77.90.31.18:4823 , piece_id=MVII6RACELENZ5VKQQRLZKAE76TIDQK543I23GBGE2SBZ6LMWHRA, seg_pos={0 0}, upload=112506]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: write tcp 5.161.89.252:40260->77.90.31.18:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:50.371678845 (128h10m38.069247181s) [seg_pos={0 0}, upload=112506]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:39:50.389652609 (128h10m38.087220946s) [upload=112506]: Done uploading segments... Dec 10 23:39:50.389674210 (128h10m38.087242546s) [upload=112506]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.020051616s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X..X.X....X.X........] duration: 362.017108ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:39:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 362.027197ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................................XXX...................] duration: 599.13181ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:39:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 599.16974ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................................XX.XX...................] duration: 469.10782ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:39:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.118321ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XXX..................] duration: 751.235637ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:39:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 751.25371ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................X..X.X....X.X.......] duration: 378.760714ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:39:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 378.781242ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.017613195s trace: https://jaeger.dev.storj.io/trace/25a0a72af01b7403 FAIL (2025-12-10 23:38:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 23:37:50.315699771 (128h8m38.013268118s) [upload=112491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:37:50.315710942 (128h8m38.013279289s) [upload=112491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:37:50.341983799 (128h8m38.039552115s) [seg_pos={0 0}, upload=112491]: Begin upload segment... Dec 10 23:37:50.342008996 (128h8m38.039577302s) [seg_pos={0 0}, upload=112491]: Done begin upload segment. Dec 10 23:37:50.342043110 (128h8m38.039611426s) [noise=true , address=77.90.31.17:4804 , piece_id=OTW4EUIOP52YVRWX5K4WNRU7YF3YRBTVA4KNRCECYQLNAIWJNLJQ, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.342063649 (128h8m38.039631965s) [noise=true , address=45.91.116.5:4307 , piece_id=BUCKWCAHZC7IOADLAWHGZRKSGQEAG25S3MJ7Q5KQ5RI5OZGHKD7Q, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.342426175 (128h8m38.039994491s) [noise=true , address=45.91.116.10:4084 , piece_id=5WKY3VXKL2U7Q74NAKPZFEM5K5DLV2ZQHCI7SEGBAUQAXVI65PDA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.342559184 (128h8m38.04012749s) [noise=true , address=145.79.227.7:4327 , piece_id=HUDQIRYDQYIDM2RTSD4Y2WQ746DKCFJJS2HQAOMDN7O5I3FVNZAA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.342810333 (128h8m38.040378649s) [noise=true , address=5.231.73.3:4032 , piece_id=TZTG24IJ7PUSQSMWA2REEXJND6AREZPTMBIXTREO6GGYOPKG6STQ, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.342829779 (128h8m38.040398085s) [noise=true , address=45.91.116.4:4950 , piece_id=OBBEDCG5OBB5VYPQIIYXJBHCLA4XWQITKMPAOE7PEWIZGSMQYT2Q, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.343047887 (128h8m38.040616203s) [noise=true , address=77.90.31.12:4613 , piece_id=64X2Y7JPPGURVRSAUW3W6YGQVEA64NRBV4KHZ7RMJ7FATJGX4VWA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.343187197 (128h8m38.040755513s) [noise=true , address=45.91.116.12:4231 , piece_id=546UH4NTFY5EHJP7GHPOTVAYEOHCA5EZMHWSVU5M56DC3TAUDFFA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.343318122 (128h8m38.040886428s) [noise=true , address=145.79.227.33:4899 , piece_id=FSHGNRSZMXBLQNCIUWI4U7QV3SRJZN4Z5L3FQMWP3VNWLCLDOHEA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.343516822 (128h8m38.041085128s) [noise=true , address=77.90.31.14:4684 , piece_id=EJ3KVQYZ3EPZDWPSVFOKT5HTWPYA6E7YWGWQJSNH5UVGIFTT4F5A, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.343600560 (128h8m38.041168866s) [noise=true , address=145.79.227.18:4040 , piece_id=3D3HU2S4V6WO3JF7A47H5OWNO7SAT4H5DFF2LNC2YFTXJDX7XKIQ, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.343830389 (128h8m38.041398695s) [noise=true , address=45.91.116.9:4281 , piece_id=ZGPWX6WOOXLZZWBVFMVGRXVXQB2JXVKWRFEFW5N6O3TGCV55MH5Q, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.343854304 (128h8m38.04142262s) [noise=true , address=77.90.31.11:4581 , piece_id=BK6EQ7OVFVKXCELQEIRHSTLEXKJI42ZZ7CPDJXGVPLUET3VHNXHA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.344130950 (128h8m38.041699266s) [noise=true , address=77.90.31.10:4536 , piece_id=L53AE7OMGFFK6MSBVZVU3MDIZWICFE2N33P2GKI7MEFOZE5T6JOQ, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.344192995 (128h8m38.041761311s) [noise=true , address=45.91.116.4:4945 , piece_id=3TVV52TTJG2ZUDGNJDXWUM6ZPPG2KCD4R4MWXX5W3NED2IOFRJ4A, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.344367221 (128h8m38.041935527s) [noise=true , address=77.90.31.16:4768 , piece_id=5WQZGMWS6EPHYR2KNSGRFGISWHHZNOUWBJ5MBMV43SFLUL3KSKKA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.344543971 (128h8m38.042112287s) [noise=true , address=45.91.116.11:4125 , piece_id=U5XDDBVNWE2DMVGQ2VWTW6HCRV7SYDI2UNJTKTVWVWA52OA455LA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.344856576 (128h8m38.042424881s) [noise=true , address=145.79.227.2:4129 , piece_id=BKAKG7H7UXLPO44PLL6J4NNXQ7MC77FRKBDHS32PACXFSDSZYQMA, seg_pos={0 0}, upload=112491]: Uploading piece... Dec 10 23:37:50.436213297 (128h8m38.133781623s) [upload=112491]: Next returned nil segment Dec 10 23:37:50.436247051 (128h8m38.133815377s) [upload=112491]: Waiting for error group managing segments... Dec 10 23:37:50.445794785 (128h8m38.143363111s) [noise=true , address=145.79.227.2:4129 , piece_id=BKAKG7H7UXLPO44PLL6J4NNXQ7MC77FRKBDHS32PACXFSDSZYQMA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.446596813 (128h8m38.14416514s) [noise=true , address=45.91.116.4:4945 , piece_id=3TVV52TTJG2ZUDGNJDXWUM6ZPPG2KCD4R4MWXX5W3NED2IOFRJ4A, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.447364847 (128h8m38.144933184s) [noise=true , address=145.79.227.7:4327 , piece_id=HUDQIRYDQYIDM2RTSD4Y2WQ746DKCFJJS2HQAOMDN7O5I3FVNZAA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.448788297 (128h8m38.146356622s) [noise=true , address=145.79.227.33:4899 , piece_id=FSHGNRSZMXBLQNCIUWI4U7QV3SRJZN4Z5L3FQMWP3VNWLCLDOHEA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.458970807 (128h8m38.156539143s) [noise=true , address=45.91.116.12:4231 , piece_id=546UH4NTFY5EHJP7GHPOTVAYEOHCA5EZMHWSVU5M56DC3TAUDFFA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.459985262 (128h8m38.157553598s) [noise=true , address=45.91.116.11:4125 , piece_id=U5XDDBVNWE2DMVGQ2VWTW6HCRV7SYDI2UNJTKTVWVWA52OA455LA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.460915570 (128h8m38.158483906s) [noise=true , address=45.91.116.4:4950 , piece_id=OBBEDCG5OBB5VYPQIIYXJBHCLA4XWQITKMPAOE7PEWIZGSMQYT2Q, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.462241968 (128h8m38.159810304s) [noise=true , address=45.91.116.9:4281 , piece_id=ZGPWX6WOOXLZZWBVFMVGRXVXQB2JXVKWRFEFW5N6O3TGCV55MH5Q, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.467818494 (128h8m38.165386831s) [noise=true , address=45.91.116.5:4307 , piece_id=BUCKWCAHZC7IOADLAWHGZRKSGQEAG25S3MJ7Q5KQ5RI5OZGHKD7Q, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.470587897 (128h8m38.168156234s) [noise=true , address=45.91.116.10:4084 , piece_id=5WKY3VXKL2U7Q74NAKPZFEM5K5DLV2ZQHCI7SEGBAUQAXVI65PDA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.542011425 (128h8m38.239579771s) [noise=true , address=145.79.227.18:4040 , piece_id=3D3HU2S4V6WO3JF7A47H5OWNO7SAT4H5DFF2LNC2YFTXJDX7XKIQ, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:37:50.548163266 (128h8m38.245731602s) [noise=true , address=5.231.73.3:4032 , piece_id=TZTG24IJ7PUSQSMWA2REEXJND6AREZPTMBIXTREO6GGYOPKG6STQ, seg_pos={0 0}, upload=112491]: Done uploading piece. err: <nil> Dec 10 23:38:50.298626815 (128h9m37.996195141s) [noise=true , address=77.90.31.10:4536 , piece_id=L53AE7OMGFFK6MSBVZVU3MDIZWICFE2N33P2GKI7MEFOZE5T6JOQ, seg_pos={0 0}, upload=112491]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:50.298700653 (128h9m37.99626896s) [noise=true , address=77.90.31.16:4768 , piece_id=5WQZGMWS6EPHYR2KNSGRFGISWHHZNOUWBJ5MBMV43SFLUL3KSKKA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: ecclient: upload failed (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF, address:77.90.31.16:4768): protocol: rpc: dial tcp 77.90.31.16:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:50.298816259 (128h9m37.996384585s) [noise=true , address=77.90.31.11:4581 , piece_id=BK6EQ7OVFVKXCELQEIRHSTLEXKJI42ZZ7CPDJXGVPLUET3VHNXHA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: rpc: dial tcp 77.90.31.11:4581: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:50.298982510 (128h9m37.996550816s) [noise=true , address=77.90.31.17:4804 , piece_id=OTW4EUIOP52YVRWX5K4WNRU7YF3YRBTVA4KNRCECYQLNAIWJNLJQ, seg_pos={0 0}, upload=112491]: Done uploading piece. err: ecclient: upload failed (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx, address:77.90.31.17:4804): protocol: rpc: dial tcp 77.90.31.17:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:50.299226906 (128h9m37.996795212s) [noise=true , address=77.90.31.14:4684 , piece_id=EJ3KVQYZ3EPZDWPSVFOKT5HTWPYA6E7YWGWQJSNH5UVGIFTT4F5A, seg_pos={0 0}, upload=112491]: Done uploading piece. err: ecclient: upload failed (node:12a7wpA6YchhigZCKC68YJzkBKwbqg1R1gBdfN8GdvrVregrK4m, address:77.90.31.14:4684): protocol: write tcp 5.161.89.252:27018->77.90.31.14:4684: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:50.299398457 (128h9m37.996966763s) [noise=true , address=77.90.31.12:4613 , piece_id=64X2Y7JPPGURVRSAUW3W6YGQVEA64NRBV4KHZ7RMJ7FATJGX4VWA, seg_pos={0 0}, upload=112491]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:38:50.299420268 (128h9m37.996988573s) [seg_pos={0 0}, upload=112491]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:38:50.314899943 (128h9m38.012468269s) [upload=112491]: Done uploading segments... Dec 10 23:38:50.314921183 (128h9m38.012489509s) [upload=112491]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.017631962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................................XX.XX..................] duration: 498.243195ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:38:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 498.263312ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................................XXX.................] duration: 672.834928ms trace: https://jaeger.dev.storj.io/trace/2b17c8a03bc718cf FAIL (2025-12-10 23:38:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 672.879152ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................X..X.X....X.X......] duration: 406.042567ms trace: https://jaeger.dev.storj.io/trace/1b146e92499c96d8 FAIL (2025-12-10 23:37:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 406.07654ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................................XX.XX.................] duration: 528.939003ms trace: https://jaeger.dev.storj.io/trace/1b53f25cc0b11955 FAIL (2025-12-10 23:37:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 529.072775ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.018169128s trace: https://jaeger.dev.storj.io/trace/31217081d93457fc FAIL (2025-12-10 23:37:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Item_ObjectBegin Dec 10 23:36:35.656095130 (128h7m23.353663446s) [upload=112474]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:36:35.679405072 (128h7m23.376973378s) [seg_pos={0 0}, upload=112474]: Begin upload segment... Dec 10 23:36:35.679433145 (128h7m23.377001461s) [seg_pos={0 0}, upload=112474]: Done begin upload segment. Dec 10 23:36:35.679451740 (128h7m23.377020056s) [noise=true , address=45.91.116.9:4274 , piece_id=NSNLVVB2GJEN7ADLGSMGURRILLGPFNZHAZ4ZC4DV32DGX4ERN4CA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.679544753 (128h7m23.377113059s) [noise=true , address=77.90.31.10:4543 , piece_id=ZRM7I4WJT2EJMUDKJY7DORN7N6QFJI7SOAO5HQLN34LRUOO7GWAA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.679995004 (128h7m23.37756331s) [noise=true , address=77.90.31.11:4576 , piece_id=WKLQAFAGOSD7YMCD2RJQTPKK7U4OAT5VPS5OXAEBRQJB7G2J7E2A, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680110460 (128h7m23.377678776s) [noise=true , address=145.79.227.44:4206 , piece_id=2Q3RN4O3VKMVQ6VTXOUPWLXUKCE5ER4D5EEETL6A3VU2SRDFQHNA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680288923 (128h7m23.377857239s) [noise=true , address=145.79.227.25:4639 , piece_id=EY2DYVHDDCYUV4XLJUNDSMZAGVHVQMLT44M7GVKLGPMMBIDD5P2A, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680517830 (128h7m23.378086137s) [noise=true , address=45.91.116.12:4256 , piece_id=6PNLP44652EJHVGEPO3EHXR6G6YDGDHFIT7KFY4D2SRFPP7ONVTA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680576681 (128h7m23.378144997s) [noise=true , address=5.231.73.5:4020 , piece_id=44WLLWHL47EUSOYX7F4QKSBPBIVPL4KUFJQXFBFT3STUSZ3PMQEA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680843670 (128h7m23.378411986s) [noise=true , address=77.90.31.12:4616 , piece_id=CSZIHDDAFW7U37QY5VFOE2VGFRDOMFUJ6TG7NUUKC6WTMWIYKSQA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.680902820 (128h7m23.378471137s) [noise=true , address=45.91.116.14:4223 , piece_id=ANIFCTKTPO3LFMMGDCUJVVSGYTNBHOYIMRNSG3A7X35T6TTLP2VA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681088006 (128h7m23.378656312s) [noise=true , address=145.79.227.30:4788 , piece_id=5UY6COIJARBN3IGEZ23Z6Q4EJXRHMCPVIQEI2UQKZAHKK2PIPT3Q, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681197941 (128h7m23.378766247s) [noise=true , address=77.90.31.13:4655 , piece_id=F5DXTRR6ZEE6WII7PXHU4DBXJR5Y26G3BIF57I4YF2S3QOWO3VLA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681413634 (128h7m23.37898199s) [noise=true , address=145.79.227.34:4931 , piece_id=JZCPQGSSAR5VGY23TRH3TNFTQCPXFKXQMZDQSC6BQFF3F65TTN4A, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681557122 (128h7m23.379125438s) [noise=true , address=77.90.31.16:4746 , piece_id=RYLC4LB75FHWGGJ5H2BLEND7DJ2U2Z3FGVPUBJT5UMDYVE5JPWCQ, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681751856 (128h7m23.379320172s) [noise=true , address=145.79.227.12:4775 , piece_id=VULPNF6Q6IM354KY4QAMCDAMQS6INXGP73IHP2GCEHUV7KZFZYNA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681848427 (128h7m23.379416743s) [noise=true , address=45.91.116.5:4300 , piece_id=WOK3AMMF3KWIM5FKNEYVSO5ZIZZ2MSCJIGYL65RIOZXBZ4LG2VZA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.681998206 (128h7m23.379566522s) [noise=true , address=77.90.31.19:4872 , piece_id=N5KG52SCRGEQ5QMZ6732NQAZJL2YQJBDFEFWKF5NKJSNZC36SAFA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.682153988 (128h7m23.379722293s) [noise=true , address=145.79.227.31:4815 , piece_id=B3T6RYWLCQEE5DWZVZMLW4C65U3WKBH55VD2B7JP4V7XO3GZJMZQ, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.682562711 (128h7m23.380131017s) [noise=true , address=145.79.227.44:4222 , piece_id=OTRFBYPUBHR3TDSMYPE3GGIC7M3A6A4VLW23MHPPN53B5GXOU2GA, seg_pos={0 0}, upload=112474]: Uploading piece... Dec 10 23:36:35.756175738 (128h7m23.453744054s) [upload=112474]: Next returned nil segment Dec 10 23:36:35.756310098 (128h7m23.453878405s) [upload=112474]: Waiting for error group managing segments... Dec 10 23:36:35.766788391 (128h7m23.464356707s) [noise=true , address=45.91.116.5:4300 , piece_id=WOK3AMMF3KWIM5FKNEYVSO5ZIZZ2MSCJIGYL65RIOZXBZ4LG2VZA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.767580251 (128h7m23.465148567s) [noise=true , address=45.91.116.9:4274 , piece_id=NSNLVVB2GJEN7ADLGSMGURRILLGPFNZHAZ4ZC4DV32DGX4ERN4CA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.767762311 (128h7m23.465330627s) [noise=true , address=45.91.116.12:4256 , piece_id=6PNLP44652EJHVGEPO3EHXR6G6YDGDHFIT7KFY4D2SRFPP7ONVTA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.770602847 (128h7m23.468171163s) [noise=true , address=145.79.227.25:4639 , piece_id=EY2DYVHDDCYUV4XLJUNDSMZAGVHVQMLT44M7GVKLGPMMBIDD5P2A, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.776445361 (128h7m23.474013677s) [noise=true , address=145.79.227.31:4815 , piece_id=B3T6RYWLCQEE5DWZVZMLW4C65U3WKBH55VD2B7JP4V7XO3GZJMZQ, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.794245652 (128h7m23.491813957s) [noise=true , address=145.79.227.34:4931 , piece_id=JZCPQGSSAR5VGY23TRH3TNFTQCPXFKXQMZDQSC6BQFF3F65TTN4A, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.794912827 (128h7m23.492481143s) [noise=true , address=145.79.227.12:4775 , piece_id=VULPNF6Q6IM354KY4QAMCDAMQS6INXGP73IHP2GCEHUV7KZFZYNA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.801998473 (128h7m23.499566789s) [noise=true , address=45.91.116.14:4223 , piece_id=ANIFCTKTPO3LFMMGDCUJVVSGYTNBHOYIMRNSG3A7X35T6TTLP2VA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:35.863391334 (128h7m23.56095965s) [noise=true , address=5.231.73.5:4020 , piece_id=44WLLWHL47EUSOYX7F4QKSBPBIVPL4KUFJQXFBFT3STUSZ3PMQEA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:36.695861673 (128h7m24.393430009s) [noise=true , address=145.79.227.44:4222 , piece_id=OTRFBYPUBHR3TDSMYPE3GGIC7M3A6A4VLW23MHPPN53B5GXOU2GA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:36.723990597 (128h7m24.421558933s) [noise=true , address=145.79.227.44:4206 , piece_id=2Q3RN4O3VKMVQ6VTXOUPWLXUKCE5ER4D5EEETL6A3VU2SRDFQHNA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:36:36.727710947 (128h7m24.425279283s) [noise=true , address=145.79.227.30:4788 , piece_id=5UY6COIJARBN3IGEZ23Z6Q4EJXRHMCPVIQEI2UQKZAHKK2PIPT3Q, seg_pos={0 0}, upload=112474]: Done uploading piece. err: <nil> Dec 10 23:37:35.639633810 (128h8m23.337202146s) [noise=true , address=77.90.31.13:4655 , piece_id=F5DXTRR6ZEE6WII7PXHU4DBXJR5Y26G3BIF57I4YF2S3QOWO3VLA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD, address:77.90.31.13:4655): protocol: rpc: dial tcp 77.90.31.13:4655: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.639787307 (128h8m23.337355642s) [noise=true , address=77.90.31.16:4746 , piece_id=RYLC4LB75FHWGGJ5H2BLEND7DJ2U2Z3FGVPUBJT5UMDYVE5JPWCQ, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: rpc: dial tcp 77.90.31.16:4746: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.639892653 (128h8m23.337460989s) [noise=true , address=77.90.31.19:4872 , piece_id=N5KG52SCRGEQ5QMZ6732NQAZJL2YQJBDFEFWKF5NKJSNZC36SAFA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL, address:77.90.31.19:4872): protocol: rpc: dial tcp 77.90.31.19:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.640142139 (128h8m23.337710475s) [noise=true , address=77.90.31.10:4543 , piece_id=ZRM7I4WJT2EJMUDKJY7DORN7N6QFJI7SOAO5HQLN34LRUOO7GWAA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 5.161.89.252:41702->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.640177455 (128h8m23.337745801s) [noise=true , address=77.90.31.11:4576 , piece_id=WKLQAFAGOSD7YMCD2RJQTPKK7U4OAT5VPS5OXAEBRQJB7G2J7E2A, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: rpc: dial tcp 77.90.31.11:4576: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.640463459 (128h8m23.338031796s) [noise=true , address=77.90.31.12:4616 , piece_id=CSZIHDDAFW7U37QY5VFOE2VGFRDOMFUJ6TG7NUUKC6WTMWIYKSQA, seg_pos={0 0}, upload=112474]: Done uploading piece. err: ecclient: upload failed (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi, address:77.90.31.12:4616): protocol: write tcp 5.161.89.252:39650->77.90.31.12:4616: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:35.640559238 (128h8m23.338127584s) [seg_pos={0 0}, upload=112474]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:37:35.655262553 (128h8m23.35283089s) [upload=112474]: Done uploading segments... Dec 10 23:37:35.655286949 (128h8m23.352855285s) [upload=112474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.018180159s core uplink us-select-1 us1-placement-20

1
...
139
140
141
142
143
...
2214