Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 121342 reports (Page 1045 of 2427)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.X.......X............] duration: 453.773708ms FAIL (2025-08-31 15:37:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.878946ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XX.XX..................] duration: 461.328612ms FAIL (2025-08-31 15:37:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 461.352888ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 742.422251ms FAIL (2025-08-31 15:37:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 742.454972ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 833.285606ms FAIL (2025-08-31 15:37:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 833.309122ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.X.......X...........] duration: 651.371032ms FAIL (2025-08-31 15:37:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 651.389848ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XX.XX.................] duration: 398.847771ms FAIL (2025-08-31 15:37:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 398.872669ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 494.818327ms FAIL (2025-08-31 15:36:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 494.844827ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XX.XX................] duration: 409.174167ms FAIL (2025-08-31 15:36:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 409.205016ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.X.......X..........] duration: 504.055647ms FAIL (2025-08-31 15:36:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.100231ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 641.833721ms FAIL (2025-08-31 15:36:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 641.857876ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 613.408565ms FAIL (2025-08-31 15:36:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 613.433854ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XX.XX...............] duration: 419.687963ms FAIL (2025-08-31 15:35:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 419.717077ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 591.109833ms FAIL (2025-08-31 15:35:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 591.142484ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.X.......X.........] duration: 435.428561ms FAIL (2025-08-31 15:35:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 435.486539ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 756.392456ms FAIL (2025-08-31 15:35:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 756.413685ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 633.199587ms FAIL (2025-08-31 15:35:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 633.217772ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XX.XX..............] duration: 616.075098ms FAIL (2025-08-31 15:35:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 616.100486ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.X.......X........] duration: 388.92847ms FAIL (2025-08-31 15:35:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 388.949199ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 623.621245ms FAIL (2025-08-31 15:34:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 623.648716ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.X.......X.......] duration: 533.084832ms FAIL (2025-08-31 15:34:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.105091ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 1.043177129s FAIL (2025-08-31 15:34:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1.04319916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 751.442524ms FAIL (2025-08-31 15:34:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 751.462281ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XX.XX.............] duration: 365.957818ms FAIL (2025-08-31 15:34:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 365.982074ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.X.......X......] duration: 361.337633ms FAIL (2025-08-31 15:33:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 361.359225ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 795.708137ms FAIL (2025-08-31 15:33:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 795.745198ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 604.746237ms FAIL (2025-08-31 15:33:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 604.764792ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.X.......X.....] duration: 644.95464ms FAIL (2025-08-31 15:33:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 644.972615ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XXXX...........] duration: 766.560531ms FAIL (2025-08-31 15:33:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 766.582372ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XX.XX............] duration: 882.476506ms FAIL (2025-08-31 15:33:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 882.508678ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XX.XX...........] duration: 615.016764ms FAIL (2025-08-31 15:32:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 615.057421ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 1.840023119s FAIL (2025-08-31 15:32:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1.840044157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 1.000659559s FAIL (2025-08-31 15:32:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.000683104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XXXX...........] duration: 729.923782ms FAIL (2025-08-31 15:32:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 729.952656ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.X.......X....] duration: 513.30375ms FAIL (2025-08-31 15:32:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 513.337754ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XX.XX..........] duration: 633.853371ms FAIL (2025-08-31 15:32:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 633.921108ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 521.856586ms FAIL (2025-08-31 15:32:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 521.883165ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 1.198041354s FAIL (2025-08-31 15:32:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.198061763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 1.234227447s FAIL (2025-08-31 15:31:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1.234254376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XX.XX.........] duration: 1.06461903s FAIL (2025-08-31 15:31:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.064641812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 2.743386268s FAIL (2025-08-31 15:31:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 2.743407979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.X.......X...] duration: 1.997357185s FAIL (2025-08-31 15:31:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.997380209s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 4.771613338s FAIL (2025-08-31 15:30:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.771639888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 8.907969116s FAIL (2025-08-31 15:30:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.907983503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XX.XX........] duration: 1.594462595s FAIL (2025-08-31 15:30:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.594491989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 4.210065898s FAIL (2025-08-31 15:30:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 4.210085615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.X.......X..] duration: 1.647688503s FAIL (2025-08-31 15:30:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.647701386s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.X.......X.] duration: 6.374720386s FAIL (2025-08-31 15:30:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.374742657s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XXXX......] duration: 1.687536208s FAIL (2025-08-31 15:29:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1.687567987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XX.XX.......] duration: 2.418743993s FAIL (2025-08-31 15:29:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments... Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment... Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment. Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece... Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments... Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil> Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments... Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 2.418772046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XXXX.....] duration: 4.241862906s FAIL (2025-08-31 15:29:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece... Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments... Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil> Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments... Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 4.24189264s core uplink us-select-1 us1-placement-20

1
...
1043
1044
1045
1046
1047
...
2427