Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX.X......X...................] duration: 503.274267ms FAIL (2025-08-16 15:55:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 503.290428ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX.........X.........X.........................] duration: 28.207052899s FAIL (2025-08-16 15:55:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.207080431s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 646.930426ms FAIL (2025-08-16 15:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 646.954781ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX.........X.........X........................] duration: 26.518432716s FAIL (2025-08-16 15:54:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.518445641s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX.X......X..................] duration: 5.982477294s FAIL (2025-08-16 15:54:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 5.982494958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX.........X.........X.......................] duration: 30.748283185s FAIL (2025-08-16 15:54:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.748298073s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 168.786979ms FAIL (2025-08-16 15:53:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 168.804592ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XX.X......X.................] duration: 738.334215ms FAIL (2025-08-16 15:53:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 738.356858ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX.........X.........X......................] duration: 35.207982307s FAIL (2025-08-16 15:53:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.208007104s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 466.181513ms FAIL (2025-08-16 15:52:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 466.202463ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX.X......X................] duration: 11.333707217s FAIL (2025-08-16 15:52:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 11.333729118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 344.949311ms FAIL (2025-08-16 15:52:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 344.976262ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 190.671569ms FAIL (2025-08-16 15:51:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 190.726092ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX.........X.........X.....................] duration: 17.445863452s FAIL (2025-08-16 15:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.445883269s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX.X......X...............] duration: 3.02291933s FAIL (2025-08-16 15:51:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 3.022943316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 411.151856ms FAIL (2025-08-16 15:51:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 411.172215ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX.X......X..............] duration: 4.649713603s FAIL (2025-08-16 15:51:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 4.649736066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 519.787452ms FAIL (2025-08-16 15:50:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 519.815824ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.................XX.........X.........X....................] duration: 14.431683252s FAIL (2025-08-16 15:50:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.431707217s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XX.X......X.............] duration: 633.541095ms FAIL (2025-08-16 15:50:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 633.563157ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 301.670107ms FAIL (2025-08-16 15:50:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 301.690956ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XX.X......X............] duration: 1.291658742s FAIL (2025-08-16 15:50:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 1.291685762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.................XX.........X.........X...................] duration: 21.887685484s FAIL (2025-08-16 15:49:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.887707475s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XX.X......X...........] duration: 1.376410741s FAIL (2025-08-16 15:49:26 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 1.376433715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 233.102669ms FAIL (2025-08-16 15:49:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 233.122515ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XX.X......X..........] duration: 1.199728066s FAIL (2025-08-16 15:48:57 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 1.199750569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.................XX.........X.........X..................] duration: 24.198520074s FAIL (2025-08-16 15:48:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.198546794s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XX.X......X.........] duration: 16.936385528s FAIL (2025-08-16 15:48:41 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 16.93640729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 387.831682ms FAIL (2025-08-16 15:48:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 387.852941ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XX.X......X........] duration: 5.10044542s FAIL (2025-08-16 15:47:59 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 5.100496154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.................XX.........X.........X.................] duration: 28.092504008s FAIL (2025-08-16 15:47:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.092522294s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 682.632028ms FAIL (2025-08-16 15:47:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 682.671251ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XX.X......X.......] duration: 9.710137698s FAIL (2025-08-16 15:46:55 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 9.71015925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.................XX.........X.........X................] duration: 15.524792442s FAIL (2025-08-16 15:46:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.524813951s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.................XX.........X.........X...............] duration: 29.664092622s FAIL (2025-08-16 15:46:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.664118119s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 297.496788ms FAIL (2025-08-16 15:46:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 297.520071ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.................XX.........X.........X..............] duration: 35.034354245s FAIL (2025-08-16 15:45:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.034378331s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XX.X......X......] duration: 1.70681941s FAIL (2025-08-16 15:45:46 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 1.706838374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XX.X......X.....] duration: 8.498563879s FAIL (2025-08-16 15:45:21 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 8.498586572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 854.464151ms FAIL (2025-08-16 15:45:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 854.485921ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XX.X......X....] duration: 785.683939ms FAIL (2025-08-16 15:44:51 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 785.714536ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 233.117086ms FAIL (2025-08-16 15:44:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 233.136782ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.................XX.........X.........X.............] duration: 26.887787142s FAIL (2025-08-16 15:44:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.887804986s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 396.631665ms FAIL (2025-08-16 15:44:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 396.659807ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.................XX.........X.........X............] duration: 38.80926874s FAIL (2025-08-16 15:44:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.809281223s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XX.X......X...] duration: 1.018579608s FAIL (2025-08-16 15:44:04 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 1.018601439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 144.556448ms FAIL (2025-08-16 15:43:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 144.582967ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.................XX.........X.........X...........] duration: 38.492225401s FAIL (2025-08-16 15:43:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:34:26.287594313 (190h8m7.018861964s) [upload=197841]: Starting upload Aug 16 15:34:26.287673282 (190h8m7.018940932s) [upload=197841]: Done starting upload Aug 16 15:34:26.287760015 (190h8m7.019027665s) [upload=197841]: Uploading segments... Aug 16 15:34:26.288099383 (190h8m7.019367033s) [upload=197841]: Got next segment. Inline: false Aug 16 15:34:26.288216152 (190h8m7.019483801s) [upload=197841]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:34:26.288222844 (190h8m7.019490494s) [upload=197841]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:26.886756649 (190h8m7.618024279s) [seg_pos={0 0}, upload=197841]: Begin upload segment... Aug 16 15:34:26.886788890 (190h8m7.61805652s) [seg_pos={0 0}, upload=197841]: Done begin upload segment. Aug 16 15:34:26.886816923 (190h8m7.618084542s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.886825118 (190h8m7.618092738s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887075188 (190h8m7.618342798s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887096758 (190h8m7.618364388s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887298477 (190h8m7.618566097s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887342901 (190h8m7.618610531s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887591587 (190h8m7.618859237s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887725810 (190h8m7.61899346s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.887983744 (190h8m7.619251395s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888213156 (190h8m7.619480806s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888377043 (190h8m7.619644694s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888639517 (190h8m7.619907166s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.888760415 (190h8m7.620028064s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889137533 (190h8m7.620405183s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889149996 (190h8m7.620417646s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889582560 (190h8m7.620850219s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889640809 (190h8m7.620908459s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:26.889990375 (190h8m7.621258025s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Uploading piece... Aug 16 15:34:27.011584069 (190h8m7.742851718s) [upload=197841]: Next returned nil segment Aug 16 15:34:27.011598355 (190h8m7.742866006s) [upload=197841]: Waiting for error group managing segments... Aug 16 15:34:27.079793204 (190h8m7.811060854s) [noise=true , address=77.90.31.16:4745 , piece_id=WV33JYRQ7UBIOCXY2VSE5RJEGV2SVSOMLWWPWV5BMGRIKWR2C37A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.121889549 (190h8m7.853157199s) [noise=true , address=77.90.31.15:4708 , piece_id=BEYFRJ3BYZBFT2V6PUUJOBCEDCQ4QXRSLSRYVV3S57APO3KI6ISA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.135396561 (190h8m7.86666421s) [noise=true , address=77.90.31.14:4665 , piece_id=PIYIU3UACZ2KWZ5PMQVEPJLFV6X4Q6YERJPATMHPGZEE52CDFNVA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.421703313 (190h8m8.152970964s) [noise=true , address=45.91.116.11:4131 , piece_id=E7XL33TLGATD6MV5DPPQWBQBSLNLEOMC4WQJN4TTLDC63GKVVU2A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:27.738756936 (190h8m8.470024566s) [noise=true , address=45.91.116.5:4307 , piece_id=P7U7YMU4QEOWOVDSX63T3QPWSZ3SFY27FQYB7OLHN55BMYFXAC4A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:28.985799854 (190h8m9.717067484s) [noise=true , address=66.23.193.175:4559 , piece_id=5DQRSCJQQJWKI5F6ZLRCDWRPE2CRVHLFEFXGGLTWGX6KFOX36PYA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:30.174181181 (190h8m10.905448851s) [noise=true , address=45.91.116.12:4237 , piece_id=EK5ZT6F3E5ZPOSIEY2IWVVPJA4WITBBZGWHLCUF7SK4MSOINKBIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:32.145934082 (190h8m12.877201732s) [noise=true , address=45.91.116.6:4999 , piece_id=SEKW2OFVU53PPBDN5JELKTONJXXCYRYW5YRXPDH76SWTWYQI2RAA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB, address:45.91.116.6:4999): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:34.486935619 (190h8m15.218203269s) [noise=true , address=79.127.220.101:4061 , piece_id=XPBWCFRKWFVQZCZM7AOUUZZDZRRHD2POGMYOAERGCWR5MEREDUKQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.408572801 (190h8m17.139840451s) [noise=true , address=104.36.82.44:4881 , piece_id=T65GDSARQ3RCE4N43E64GOVGWBQOK4XKGGQMUAHSQTWNPIGOGVSQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:36.892474253 (190h8m17.623741903s) [noise=true , address=77.90.31.18:4822 , piece_id=4XFI6KEYMW7B5OOK6AZ3DELBPQSQ6L6YCNAS6MUG6PU6BSP2PGIQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:41.598275361 (190h8m22.32954301s) [noise=true , address=104.36.82.58:4266 , piece_id=Q6BIUXOPA72Q2FUO2HJWMBTRYFDF5EJUOAI2XIXHRWFGVBPPLY6Q, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:43.463190476 (190h8m24.194458126s) [noise=true , address=104.36.82.2:4752 , piece_id=3XZP235LZRTP2OHICT7LJTZNDVECHHBO4W2QEWKK23VFP6MK7JJQ, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:34:44.381727093 (190h8m25.112994743s) [noise=true , address=104.36.82.60:4351 , piece_id=CUSUZ7DSMOJXMCUNR4GUDS6AU47PKKQIRSOCCFNYXOWJNVNDZ6RA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:02.431209734 (190h8m43.162477384s) [noise=true , address=104.36.82.23:4479 , piece_id=2P3BCTIHTKEKJ3DIEIBAJ5FREH6Q4PS6DZFGUN5BOADNZFNFMOXA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: <nil> Aug 16 15:35:26.269944636 (190h9m7.001212286s) [noise=true , address=104.36.82.5:4853 , piece_id=D46JRZGEJMRLK3BY2266LHZUHQB7XIZAFS3SUSMQWAEWBCXVQ6TA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:19100->104.36.82.5:4853: use of closed network connection Aug 16 15:35:26.270108384 (190h9m7.001376034s) [noise=true , address=77.90.31.11:4568 , piece_id=PYUENPYYDKRPTHF4GHVIRT6ABQLLMW2MBNP7PIXPU7OMBWPDV23A, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:26.270127429 (190h9m7.00139509s) [noise=true , address=77.90.31.19:4848 , piece_id=C4EQUIQABL63YSTLI5MTTR35MU33VQQUSOHIY2EOJ4B425KO6THA, seg_pos={0 0}, upload=197841]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 5.161.89.252:45938->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:26.270152938 (190h9m7.001420587s) [seg_pos={0 0}, upload=197841]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:35:27.945901312 (190h9m8.677169012s) [upload=197841]: Done uploading segments... Aug 16 15:35:27.945924786 (190h9m8.677192416s) [upload=197841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.4922465s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 132.155675ms FAIL (2025-08-16 15:43:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after add180cd317e136e9a7269306da283467a21eadd86dcadb7122496ee78413e50 != before b700f26ab6ceef98f22cf7e1ab637c0c58f804be339381b2834069199fdb5975 ======LOGS====== Aug 16 15:21:20.817605457 (189h55m1.548873088s) [upload=197614]: Starting upload Aug 16 15:21:20.817644491 (189h55m1.548912111s) [upload=197614]: Done starting upload Aug 16 15:21:20.817690889 (189h55m1.548958508s) [upload=197614]: Uploading segments... Aug 16 15:21:20.817726545 (189h55m1.548994165s) [upload=197614]: Got next segment. Inline: true Aug 16 15:21:20.817730163 (189h55m1.548997793s) [upload=197614]: Waiting for error group managing segments... Aug 16 15:21:20.817763114 (189h55m1.549030744s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:20.817766591 (189h55m1.54903421s) [upload=197614]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:20.817769385 (189h55m1.549037016s) [upload=197614]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.010025302 (189h55m12.741292962s) [upload=197614]: Done uploading segments... Aug 16 15:21:32.010051571 (189h55m12.741319231s) [upload=197614]: Upload finished. err: <nil> Duration:* 132.176274ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XX.X......X..] duration: 1.058413805s FAIL (2025-08-16 15:42:55 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:41:12.402104593 (190h14m53.133372223s) [upload=197963]: Starting upload Aug 16 15:41:12.402160428 (190h14m53.133428058s) [upload=197963]: Done starting upload Aug 16 15:41:12.402246779 (190h14m53.1335144s) [upload=197963]: Uploading segments... Aug 16 15:41:12.402480159 (190h14m53.133747789s) [upload=197963]: Got next segment. Inline: false Aug 16 15:41:12.402508461 (190h14m53.133776082s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:41:12.402512469 (190h14m53.133780089s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:41:12.432979783 (190h14m53.164247433s) [seg_pos={0 0}, upload=197963]: Begin upload segment... Aug 16 15:41:12.433069851 (190h14m53.164337502s) [seg_pos={0 0}, upload=197963]: Done begin upload segment. Aug 16 15:41:12.433300475 (190h14m53.164568125s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.433371609 (190h14m53.164639269s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.434684907 (190h14m53.165952566s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.435481013 (190h14m53.166748662s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.436348603 (190h14m53.167616253s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.437054379 (190h14m53.16832203s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.440286702 (190h14m53.171554363s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.445732877 (190h14m53.177000497s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.447246201 (190h14m53.178513851s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.448284202 (190h14m53.179551832s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.449463608 (190h14m53.180731238s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450094433 (190h14m53.181362063s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450145950 (190h14m53.18141357s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.450200563 (190h14m53.181468183s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451444931 (190h14m53.182712551s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.451774661 (190h14m53.18304228s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Uploading piece... Aug 16 15:41:12.534048827 (190h14m53.265316477s) [upload=197963]: Next returned nil segment Aug 16 15:41:12.534088210 (190h14m53.265355861s) [upload=197963]: Waiting for error group managing segments... Aug 16 15:41:12.548971298 (190h14m53.280238948s) [noise=true , address=104.36.82.15:4257 , piece_id=H4FXNRR2DVAZB2UZBJTMMDSDQK5KIVH7BUZQRFR3ARQS5SKHYATA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.550856190 (190h14m53.28212381s) [noise=true , address=104.36.82.7:4910 , piece_id=SJNRRSBANKU45SJVFTIX2CV5S2QQE7YYZPKWLS7DPE5D4SWZCVPA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.590344154 (190h14m53.321611804s) [noise=true , address=104.36.82.6:4876 , piece_id=ZXYRDJ47L5C2E7JCYNSMXNN3OQGWGTJZOK4H36OTGXBILOWQGMYA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.601114663 (190h14m53.332382293s) [noise=true , address=104.36.82.12:4151 , piece_id=VBGUSFCI6M6ZPHVEJ5NUE64VJGZTVKODJXXKCUIWK6ZSYAPKXXGA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:12.823455396 (190h14m53.554723036s) [noise=true , address=77.90.31.15:4698 , piece_id=T2YHKVVZNSWEPXDBEQHLGIJTH2FBEIUXIOJ2F3XOZ64BRRCZ6HJA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.416489395 (190h14m54.147757015s) [noise=true , address=45.91.116.6:4974 , piece_id=KDFOS7U7JUILPHQGL5TTARYUYCU67I7A2VF52YMQG6RHKKJWP3QA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:13.532032808 (190h14m54.263300438s) [noise=true , address=45.91.116.11:4119 , piece_id=EP427WIGG7C52GF5KCPTZLI5H3GVGC5FOM5YJLZNOGSDADLWFK5Q, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.035449985 (190h14m54.766717605s) [noise=true , address=77.90.31.13:4626 , piece_id=O4JLXXU6VJVMBDO2C6PTTGBAP2XPN7F6H44WVKE7YA2M2VGKKHAQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.103439759 (190h14m54.834707389s) [noise=true , address=45.91.116.4:4940 , piece_id=IPI3YEWLAZZYSJJWE2Y3Y6TWUGXSAVUFJGUVHBMZAB7PL4RQHCXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.347174953 (190h14m55.078442583s) [noise=true , address=45.91.116.13:4155 , piece_id=K7DWZJUFF4A54KTYZK6GDQV5MQTRWL6CQHHC7AO6S6HM5UQGYDBA, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:14.439594001 (190h14m55.170861631s) [noise=true , address=45.91.116.12:4227 , piece_id=TIEX2JV7TULV73J6ECGWWF4IRJ5T7NMRT72DTGPFXCS2LOWFMZXQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:36.322085073 (190h15m17.053352703s) [noise=true , address=104.36.82.3:4767 , piece_id=3QTVTENB4N57OK4LXIRGEQ3GQ5L2MB5PVSIWCWRO5EQJ2JIREGKQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072321103 (190h15m21.803588722s) [noise=true , address=104.36.82.16:4293 , piece_id=TND7VYCXYY3WKHOTZA3KEBSBCJQAGVUW3I5GXN54DWNG2NH3WNZQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: <nil> Aug 16 15:41:41.072335469 (190h15m21.803603099s) [seg_pos={0 0}, upload=197963]: Segment reached optimal threshold of 13 pieces. Aug 16 15:41:41.072678443 (190h15m21.803946073s) [noise=true , address=77.90.31.21:4878 , piece_id=YYKY7ME3GAYPYFRY4G4LWDYOB63WPPRB5QZXFHGIIUDCZL73GKMQ, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: rpc: dial tcp 77.90.31.21:4878: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072720493 (190h15m21.803988123s) [noise=true , address=104.36.82.9:4007 , piece_id=CDZBRNZBS3NHADILYDNG3XL6MNUGA6BP3EV3YJHLRSEITS2PIB6A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:39758->104.36.82.9:4007: use of closed network connection Aug 16 15:41:41.072750890 (190h15m21.804018511s) [noise=true , address=77.90.31.21:4877 , piece_id=ZET5MRUHU4RPHN6HPF7HR3LHJD62T5HPA56UQZTTUD2NWKM7TJ7A, seg_pos={0 0}, upload=197963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 5.161.89.252:12140->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:41:41.072758936 (190h15m21.804026566s) [seg_pos={0 0}, upload=197963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:41:41.072806925 (190h15m21.804074555s) [upload=197963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:41:41.072810192 (190h15m21.804077812s) [upload=197963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:41:41.249494107 (190h15m21.980761756s) [upload=197963]: Done uploading segments... Aug 16 15:41:41.249514585 (190h15m21.980782235s) [upload=197963]: Upload finished. err: <nil> Duration: 1.058431807s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)