Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.............X.....] duration: 4.001514182s FAIL (2025-08-16 16:24:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 16:21:19.736330105 (190h55m0.467597755s) [upload=198661]: Starting upload Aug 16 16:21:19.736403663 (190h55m0.467671313s) [upload=198661]: Done starting upload Aug 16 16:21:19.736486540 (190h55m0.467754189s) [upload=198661]: Uploading segments... Aug 16 16:21:19.736523800 (190h55m0.467791449s) [upload=198661]: Got next segment. Inline: true Aug 16 16:21:19.736529099 (190h55m0.467796749s) [upload=198661]: Waiting for error group managing segments... Aug 16 16:21:19.736572000 (190h55m0.467839651s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:21:19.736577090 (190h55m0.467844741s) [upload=198661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:21:19.736580446 (190h55m0.467848097s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:21:19.820858302 (190h55m0.552125953s) [upload=198661]: Done uploading segments... Aug 16 16:21:19.820875544 (190h55m0.552143195s) [upload=198661]: Upload finished. err: metaclient: Too Many Requests Duration:* 4.001548655s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 362.414579ms FAIL (2025-08-16 16:23:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 362.450727ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.............X....] duration: 7.058718071s FAIL (2025-08-16 16:23:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 16:21:19.736330105 (190h55m0.467597755s) [upload=198661]: Starting upload Aug 16 16:21:19.736403663 (190h55m0.467671313s) [upload=198661]: Done starting upload Aug 16 16:21:19.736486540 (190h55m0.467754189s) [upload=198661]: Uploading segments... Aug 16 16:21:19.736523800 (190h55m0.467791449s) [upload=198661]: Got next segment. Inline: true Aug 16 16:21:19.736529099 (190h55m0.467796749s) [upload=198661]: Waiting for error group managing segments... Aug 16 16:21:19.736572000 (190h55m0.467839651s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:21:19.736577090 (190h55m0.467844741s) [upload=198661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:21:19.736580446 (190h55m0.467848097s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:21:19.820858302 (190h55m0.552125953s) [upload=198661]: Done uploading segments... Aug 16 16:21:19.820875544 (190h55m0.552143195s) [upload=198661]: Upload finished. err: metaclient: Too Many Requests Duration:* 7.058750211s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 454.655526ms FAIL (2025-08-16 16:23:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 454.679321ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.............X...] duration: 7.239525845s FAIL (2025-08-16 16:22:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 16:21:19.736330105 (190h55m0.467597755s) [upload=198661]: Starting upload Aug 16 16:21:19.736403663 (190h55m0.467671313s) [upload=198661]: Done starting upload Aug 16 16:21:19.736486540 (190h55m0.467754189s) [upload=198661]: Uploading segments... Aug 16 16:21:19.736523800 (190h55m0.467791449s) [upload=198661]: Got next segment. Inline: true Aug 16 16:21:19.736529099 (190h55m0.467796749s) [upload=198661]: Waiting for error group managing segments... Aug 16 16:21:19.736572000 (190h55m0.467839651s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:21:19.736577090 (190h55m0.467844741s) [upload=198661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:21:19.736580446 (190h55m0.467848097s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:21:19.820858302 (190h55m0.552125953s) [upload=198661]: Done uploading segments... Aug 16 16:21:19.820875544 (190h55m0.552143195s) [upload=198661]: Upload finished. err: metaclient: Too Many Requests Duration:* 7.239558186s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.372596578s FAIL (2025-08-16 16:22:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.372623128s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 539.610618ms FAIL (2025-08-16 16:22:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 539.636887ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.............X..] duration: 5.785016312s FAIL (2025-08-16 16:22:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 16:21:19.736330105 (190h55m0.467597755s) [upload=198661]: Starting upload Aug 16 16:21:19.736403663 (190h55m0.467671313s) [upload=198661]: Done starting upload Aug 16 16:21:19.736486540 (190h55m0.467754189s) [upload=198661]: Uploading segments... Aug 16 16:21:19.736523800 (190h55m0.467791449s) [upload=198661]: Got next segment. Inline: true Aug 16 16:21:19.736529099 (190h55m0.467796749s) [upload=198661]: Waiting for error group managing segments... Aug 16 16:21:19.736572000 (190h55m0.467839651s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:21:19.736577090 (190h55m0.467844741s) [upload=198661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:21:19.736580446 (190h55m0.467848097s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:21:19.820858302 (190h55m0.552125953s) [upload=198661]: Done uploading segments... Aug 16 16:21:19.820875544 (190h55m0.552143195s) [upload=198661]: Upload finished. err: metaclient: Too Many Requests Duration:* 5.785044083s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 4.154145343s FAIL (2025-08-16 16:22:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.154295095s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.............X.] duration: 5.73051036s FAIL (2025-08-16 16:21:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 16:21:19.736330105 (190h55m0.467597755s) [upload=198661]: Starting upload Aug 16 16:21:19.736403663 (190h55m0.467671313s) [upload=198661]: Done starting upload Aug 16 16:21:19.736486540 (190h55m0.467754189s) [upload=198661]: Uploading segments... Aug 16 16:21:19.736523800 (190h55m0.467791449s) [upload=198661]: Got next segment. Inline: true Aug 16 16:21:19.736529099 (190h55m0.467796749s) [upload=198661]: Waiting for error group managing segments... Aug 16 16:21:19.736572000 (190h55m0.467839651s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:21:19.736577090 (190h55m0.467844741s) [upload=198661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:21:19.736580446 (190h55m0.467848097s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:21:19.820858302 (190h55m0.552125953s) [upload=198661]: Done uploading segments... Aug 16 16:21:19.820875544 (190h55m0.552143195s) [upload=198661]: Upload finished. err: metaclient: Too Many Requests Duration:* 5.730533082s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......X....................................................] duration: 461.364823ms FAIL (2025-08-16 16:21:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 461.388908ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.............X] duration: 99.076632ms FAIL (2025-08-16 16:21:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 16:21:19.736330105 (190h55m0.467597755s) [upload=198661]: Starting upload Aug 16 16:21:19.736403663 (190h55m0.467671313s) [upload=198661]: Done starting upload Aug 16 16:21:19.736486540 (190h55m0.467754189s) [upload=198661]: Uploading segments... Aug 16 16:21:19.736523800 (190h55m0.467791449s) [upload=198661]: Got next segment. Inline: true Aug 16 16:21:19.736529099 (190h55m0.467796749s) [upload=198661]: Waiting for error group managing segments... Aug 16 16:21:19.736572000 (190h55m0.467839651s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:21:19.736577090 (190h55m0.467844741s) [upload=198661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:21:19.736580446 (190h55m0.467848097s) [upload=198661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:21:19.820858302 (190h55m0.552125953s) [upload=198661]: Done uploading segments... Aug 16 16:21:19.820875544 (190h55m0.552143195s) [upload=198661]: Upload finished. err: metaclient: Too Many Requests Duration:* 99.099545ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 4.217372051s FAIL (2025-08-16 16:21:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.217402538s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......X...................................................] duration: 611.46133ms FAIL (2025-08-16 16:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 611.480506ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X......X..................................................] duration: 647.966405ms FAIL (2025-08-16 16:20:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 647.995499ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.X......X.................................................] duration: 397.510311ms FAIL (2025-08-16 16:20:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 397.626411ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.971711908s FAIL (2025-08-16 16:20:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.971740192s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.84635457s FAIL (2025-08-16 16:20:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 5.846375561s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.544903491s FAIL (2025-08-16 16:19:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 5.544935801s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX.X......X................................................] duration: 381.936979ms FAIL (2025-08-16 16:19:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 381.989787ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 10.149730755s FAIL (2025-08-16 16:19:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.149755071s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX.X......X...............................................] duration: 400.051108ms FAIL (2025-08-16 16:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 400.078699ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX.X......X..............................................] duration: 395.990089ms FAIL (2025-08-16 16:18:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 396.013553ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.50075656s FAIL (2025-08-16 16:18:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.500775445s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.433717284s FAIL (2025-08-16 16:18:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 6.433742973s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.134907698s FAIL (2025-08-16 16:17:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.134927497s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX.X......X.............................................] duration: 396.175177ms FAIL (2025-08-16 16:17:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 396.191218ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.255296708s FAIL (2025-08-16 16:17:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 6.255310885s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX.X......X............................................] duration: 469.269235ms FAIL (2025-08-16 16:16:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 469.287339ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 7.193346767s FAIL (2025-08-16 16:16:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.193380992s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.893327095s FAIL (2025-08-16 16:16:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 6.893349678s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX.X......X...........................................] duration: 620.681756ms FAIL (2025-08-16 16:16:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 620.70519ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 8.684282701s FAIL (2025-08-16 16:15:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.68430896s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.54488486s FAIL (2025-08-16 16:15:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 5.544902633s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.110183783s FAIL (2025-08-16 16:15:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.110223367s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 9.883706144s FAIL (2025-08-16 16:15:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 9.883725569s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX.X......X..........................................] duration: 449.134584ms FAIL (2025-08-16 16:15:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 449.158359ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.........X.................................................] duration: 8.285528449s FAIL (2025-08-16 16:15:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.285544489s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.........X................................................] duration: 12.896494462s FAIL (2025-08-16 16:14:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.896519288s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.624135418s FAIL (2025-08-16 16:14:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 6.624179511s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 8.465345668s FAIL (2025-08-16 16:14:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 8.465374232s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX.X......X.........................................] duration: 384.400866ms FAIL (2025-08-16 16:13:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 384.417207ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 12.152601012s FAIL (2025-08-16 16:13:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 12.152629867s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.........X...............................................] duration: 5.606747691s FAIL (2025-08-16 16:13:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.606784129s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX.X......X........................................] duration: 356.762948ms FAIL (2025-08-16 16:12:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 356.791053ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.........X..............................................] duration: 6.889852078s FAIL (2025-08-16 16:12:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.889875332s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.........X.............................................] duration: 12.560505705s FAIL (2025-08-16 16:12:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.560520875s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 9.163921004s FAIL (2025-08-16 16:12:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 9.163952844s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX.X......X.......................................] duration: 367.382734ms FAIL (2025-08-16 16:11:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 367.406668ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 13.549688765s FAIL (2025-08-16 16:11:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f36581b1eb373626007067365b048f5ce5cc5372ae2c12cedc5ebb9fac1fe3b1 != before 5103a32e94533b210bafbc980191e5d65541c62ead63349960a29c0fe846ecfc ======LOGS====== Aug 16 16:10:27.681038132 (190h44m8.412305783s) [upload=198465]: Starting upload Aug 16 16:10:27.681106641 (190h44m8.412374291s) [upload=198465]: Done starting upload Aug 16 16:10:27.681180781 (190h44m8.412448431s) [upload=198465]: Uploading segments... Aug 16 16:10:27.681239091 (190h44m8.41250674s) [upload=198465]: Got next segment. Inline: true Aug 16 16:10:27.681246394 (190h44m8.412514045s) [upload=198465]: Waiting for error group managing segments... Aug 16 16:10:27.681294365 (190h44m8.412562016s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:10:27.681300507 (190h44m8.412568157s) [upload=198465]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:10:27.681304214 (190h44m8.412571864s) [upload=198465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:10:44.137637975 (190h44m24.868905625s) [upload=198465]: Done uploading segments... Aug 16 16:10:44.137657582 (190h44m24.868925242s) [upload=198465]: Upload finished. err: <nil> Duration:* 13.549721187s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)