Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 121393 reports (Page 1043 of 2428)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.XX........................................................] duration: 365.878604ms FAIL (2025-08-31 16:08:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 365.900025ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration: 667.804307ms FAIL (2025-08-31 16:08:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 667.825248ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X.......X.................................................] duration: 370.882265ms FAIL (2025-08-31 16:07:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 370.908433ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 693.01914ms FAIL (2025-08-31 16:07:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 693.055288ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.XX.......................................................] duration: 356.708709ms FAIL (2025-08-31 16:07:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 356.7359ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..XXXX......................................................] duration: 596.716952ms FAIL (2025-08-31 16:07:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 596.734336ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 387.575832ms FAIL (2025-08-31 16:07:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 387.593826ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...XXXX.....................................................] duration: 656.962695ms FAIL (2025-08-31 16:06:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 656.995917ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.X.......X................................................] duration: 546.581458ms FAIL (2025-08-31 16:06:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 546.604391ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX.XX......................................................] duration: 361.608372ms FAIL (2025-08-31 16:06:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 361.629683ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration: 419.252532ms FAIL (2025-08-31 16:06:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 419.279232ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.X.......X...............................................] duration: 592.41629ms FAIL (2025-08-31 16:06:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 592.436038ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....XXXX....................................................] duration: 1.183798226s FAIL (2025-08-31 16:05:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.183981741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX.XX.....................................................] duration: 439.749547ms FAIL (2025-08-31 16:05:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 439.80493ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XXXX......................................................] duration: 614.020261ms FAIL (2025-08-31 16:05:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 614.047081ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.X.......X..............................................] duration: 440.373381ms FAIL (2025-08-31 16:05:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 440.396315ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX.XX....................................................] duration: 443.250107ms FAIL (2025-08-31 16:04:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 443.269394ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....XXXX...................................................] duration: 619.574921ms FAIL (2025-08-31 16:04:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 619.606701ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XXXX.....................................................] duration: 711.436235ms FAIL (2025-08-31 16:04:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 711.458857ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.X.......X.............................................] duration: 355.325234ms FAIL (2025-08-31 16:04:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 355.342445ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......XXXX..................................................] duration: 627.075312ms FAIL (2025-08-31 16:03:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 627.103545ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX.XX...................................................] duration: 519.472499ms FAIL (2025-08-31 16:03:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 519.495242ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......XXXX.................................................] duration: 733.374659ms FAIL (2025-08-31 16:03:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 733.395159ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XXXX....................................................] duration: 636.452758ms FAIL (2025-08-31 16:03:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 636.481882ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.X.......X............................................] duration: 436.887389ms FAIL (2025-08-31 16:03:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 436.910121ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........XXXX................................................] duration: 648.942984ms FAIL (2025-08-31 16:03:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 648.961629ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX.XX..................................................] duration: 358.985783ms FAIL (2025-08-31 16:02:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 359.02614ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XXXX...................................................] duration: 639.058403ms FAIL (2025-08-31 16:02:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 639.098199ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.X.......X...........................................] duration: 618.19923ms FAIL (2025-08-31 16:02:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 618.228216ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX.XX.................................................] duration: 392.809609ms FAIL (2025-08-31 16:02:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 392.83148ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........XXXX...............................................] duration: 833.46953ms FAIL (2025-08-31 16:02:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 833.486673ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.X.......X..........................................] duration: 426.194664ms FAIL (2025-08-31 16:01:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.214803ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XXXX..................................................] duration: 613.304374ms FAIL (2025-08-31 16:01:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 613.350621ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XXXX.................................................] duration: 688.046225ms FAIL (2025-08-31 16:01:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 688.079487ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX.XX................................................] duration: 558.417913ms FAIL (2025-08-31 16:01:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 558.438814ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........XXXX..............................................] duration: 708.529884ms FAIL (2025-08-31 16:01:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 708.710575ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.X.......X.........................................] duration: 411.444796ms FAIL (2025-08-31 16:00:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 411.459854ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XXXX................................................] duration: 472.050242ms FAIL (2025-08-31 16:00:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 472.077684ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX.XX...............................................] duration: 338.825718ms FAIL (2025-08-31 16:00:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 338.877857ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........XXXX.............................................] duration: 715.559146ms FAIL (2025-08-31 16:00:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 715.572232ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX.XX..............................................] duration: 753.26479ms FAIL (2025-08-31 15:59:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 753.294987ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XXXX...............................................] duration: 876.489779ms FAIL (2025-08-31 15:59:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 876.545733ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.X.......X........................................] duration: 413.341085ms FAIL (2025-08-31 15:59:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 413.424313ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.X.......X.......................................] duration: 407.180839ms FAIL (2025-08-31 15:59:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 407.199013ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX.XX.............................................] duration: 518.381083ms FAIL (2025-08-31 15:59:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 518.399237ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XXXX..............................................] duration: 467.526186ms FAIL (2025-08-31 15:59:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 467.564427ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............XXXX............................................] duration: 629.549789ms FAIL (2025-08-31 15:59:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 629.571099ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XXXX.............................................] duration: 427.393636ms FAIL (2025-08-31 15:58:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 427.405419ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.X.......X......................................] duration: 399.090057ms FAIL (2025-08-31 15:58:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 399.115505ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XX.XX............................................] duration: 377.708418ms FAIL (2025-08-31 15:58:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 377.737352ms core uplink us-select-1

1
...
1041
1042
1043
1044
1045
...
2428