Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration:
798.858109ms
FAIL (2025-08-31 15:49:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 798.883206ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration:
977.274696ms
FAIL (2025-08-31 15:49:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 977.300244ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................XX.XX.................................] duration:
392.150511ms
FAIL (2025-08-31 15:49:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 392.176381ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X.X.......X...........................] duration:
389.279473ms
FAIL (2025-08-31 15:48:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 389.297467ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration:
912.870891ms
FAIL (2025-08-31 15:48:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 912.889587ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X.X.......X..........................] duration:
370.931291ms
FAIL (2025-08-31 15:48:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 370.955206ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................XXXX................................] duration:
408.171825ms
FAIL (2025-08-31 15:48:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 408.197732ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................XX.XX................................] duration:
399.398818ms
FAIL (2025-08-31 15:48:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 399.422542ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration:
668.057191ms
FAIL (2025-08-31 15:47:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 668.086065ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................XX.XX...............................] duration:
299.213821ms
FAIL (2025-08-31 15:47:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 299.249078ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................X.X.......X.........................] duration:
358.288887ms
FAIL (2025-08-31 15:47:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 358.307522ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [............................XXXX............................] duration:
959.458804ms
FAIL (2025-08-31 15:47:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 959.47259ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................XX.XX..............................] duration:
429.003866ms
FAIL (2025-08-31 15:47:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 429.026358ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration:
489.784149ms
FAIL (2025-08-31 15:47:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 489.808977ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................X.X.......X........................] duration:
432.950097ms
FAIL (2025-08-31 15:47:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 432.969974ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration:
813.831208ms
FAIL (2025-08-31 15:46:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 813.857898ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................XX.XX.............................] duration:
489.294461ms
FAIL (2025-08-31 15:46:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 489.310801ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration:
565.42035ms
FAIL (2025-08-31 15:46:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 565.470725ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................X.X.......X.......................] duration:
394.165885ms
FAIL (2025-08-31 15:46:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 394.18979ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration:
806.620502ms
FAIL (2025-08-31 15:45:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 806.653723ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................XX.XX............................] duration:
380.448597ms
FAIL (2025-08-31 15:45:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 380.498161ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration:
586.278465ms
FAIL (2025-08-31 15:45:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 586.359516ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.X.......X......................] duration:
430.333222ms
FAIL (2025-08-31 15:45:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 430.364029ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................XXXX............................] duration:
643.620321ms
FAIL (2025-08-31 15:44:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 643.679573ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................XX.XX...........................] duration:
356.194411ms
FAIL (2025-08-31 15:44:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 356.212275ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration:
745.797168ms
FAIL (2025-08-31 15:44:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 745.819049ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...............................XXXX.........................] duration:
1.023652567s
FAIL (2025-08-31 15:44:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.023678744s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................X.X.......X.....................] duration:
658.807917ms
FAIL (2025-08-31 15:44:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 658.830771ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration:
509.835055ms
FAIL (2025-08-31 15:43:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 509.856064ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................XX.XX..........................] duration:
439.478488ms
FAIL (2025-08-31 15:43:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 439.501893ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................X.X.......X....................] duration:
433.142696ms
FAIL (2025-08-31 15:43:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 433.161383ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [................................XXXX........................] duration:
680.278603ms
FAIL (2025-08-31 15:43:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 680.300053ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................XXXX.........................] duration:
496.242301ms
FAIL (2025-08-31 15:43:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 496.298809ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................XX.XX.........................] duration:
707.774761ms
FAIL (2025-08-31 15:43:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 707.805219ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................XX.XX........................] duration:
395.231089ms
FAIL (2025-08-31 15:42:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 395.253762ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................X.X.......X...................] duration:
366.931548ms
FAIL (2025-08-31 15:42:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 366.949382ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration:
646.929321ms
FAIL (2025-08-31 15:42:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 646.946373ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................XXXX........................] duration:
764.016058ms
FAIL (2025-08-31 15:42:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 764.046105ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................X.X.......X..................] duration:
470.849997ms
FAIL (2025-08-31 15:41:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 470.869854ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................XX.XX.......................] duration:
662.876545ms
FAIL (2025-08-31 15:41:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 662.903805ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration:
1.21611228s
FAIL (2025-08-31 15:41:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 1.216134741s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration:
913.359721ms
FAIL (2025-08-31 15:41:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 913.378476ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration:
706.995322ms
FAIL (2025-08-31 15:40:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 707.014196ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................X.X.......X.................] duration:
643.142136ms
FAIL (2025-08-31 15:40:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 643.167645ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................XX.XX......................] duration:
376.343412ms
FAIL (2025-08-31 15:40:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 376.365333ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X.X.......X................] duration:
608.277656ms
FAIL (2025-08-31 15:40:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 608.29517ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration:
423.280198ms
FAIL (2025-08-31 15:40:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
46:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724433131 (49h16m28.07721826s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724443612 (49h16m28.07722872s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724839897 (49h16m28.077625005s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.724964822 (49h16m28.077749941s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725146435 (49h16m28.077931543s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725409811 (49h16m28.078194929s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725427805 (49h16m28.078212913s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725739501 (49h16m28.078524609s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.725879655 (49h16m28.078664783s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726029477 (49h16m28.078814585s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.726570105 (49h16m28.079355233s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727036443 (49h16m28.079821572s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.727643476 (49h16m28.080428604s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.730666428 (49h16m28.083451536s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Uploading piece...
Aug 31 15:24:52.813733607 (49h16m28.166518705s) [upload=43391]: Next returned nil segment
Aug 31 15:24:52.813745710 (49h16m28.166530807s) [upload=43391]: Waiting for error group managing segments...
Aug 31 15:24:52.821752915 (49h16m28.174538013s) [noise=true , address=77.90.31.17:4774 , piece_id=WTDYK7FB4EPG5AH7DDW7WWOPLNLBMXWTUUZYM2AHJ5G32ISOHJSA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.821837465 (49h16m28.174622562s) [noise=true , address=45.91.116.4:4961 , piece_id=ZZ3ALIFV2OHLPB3QLSPLOEWR6LNGBS7UZ6W3WI75ZVCACJPW2XLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.822609347 (49h16m28.175394445s) [noise=true , address=77.90.31.18:4824 , piece_id=5OFIRVBZEKVZKUP3RGCVLGCLYZNLUBEBULH72XVL3S3LRND6SP5A, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.825449756 (49h16m28.178234854s) [noise=true , address=77.90.31.16:4746 , piece_id=ETAAVFUN55UDZM4RHH6MAZEOLR5LFJWBUT324GYEOSICEAVSWSHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:52.858786902 (49h16m28.21157203s) [noise=true , address=77.90.31.13:4633 , piece_id=GQPQUAEYIEE4DTJ3NGROI3ZGQP3KE6XNA2QXIAI6ISPOV5XWM6AA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.251874255 (49h16m28.604659363s) [noise=true , address=79.127.205.246:4281 , piece_id=Y3SJ27YAPK5V2ND3KU2NVECIB73HR3AAJJIY73R2L4FNUYQURQHQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:24:53.262681142 (49h16m28.61546625s) [noise=true , address=79.127.205.245:4226 , piece_id=WTRZBWI7QXYIV7REFPBMEEKXRES2X4T3HZXTX3OOKIZQR2NDXKBA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: <nil>
Aug 31 15:25:52.692636900 (49h17m28.045422028s) [noise=true , address=104.36.82.60:4340 , piece_id=5B2CQMTZSU7AX4QA7EXYOTLHMCRDIBVY7ADAQBGHUVOXK33RW4IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692732950 (49h17m28.045518078s) [noise=true , address=104.36.82.42:4811 , piece_id=OZL6YAGBZAAIFXMZY4CNG3ZXTUCKPKE3OBVJWDVDE66IRLL4H2IA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: rpc: dial tcp 104.36.82.42:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692770521 (49h17m28.04555565s) [noise=true , address=104.36.82.4:4831 , piece_id=S4B6GO2KUKYMS3X6Y4QJZKBZ37QJGVE4FZP7AHTIPNADGQRAK4RQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692790268 (49h17m28.045575397s) [noise=true , address=104.36.82.53:4132 , piece_id=A2SN4O6LWOXO7MU75R5M2JADQCCF7KR43KKFUKG5LWOF76YLFXOA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: rpc: dial tcp 104.36.82.53:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692837236 (49h17m28.045622354s) [noise=true , address=104.36.82.44:4898 , piece_id=PFFMWXHP2EKSI3JNAWUICKTZTLIRZF35CWBX4KGJGKML5GNDNHLQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692863005 (49h17m28.045648133s) [noise=true , address=104.36.82.23:4453 , piece_id=AE4IGEKFE237RO3HMZQO6R4G4C2U2RBC2W54KE2VUKTZ4VZFA2DQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692895607 (49h17m28.045680735s) [noise=true , address=104.36.82.7:4924 , piece_id=Y6XO5V4WD6GYIPIT7JBFWYNM6GV5PKCEI7H5XV57B7NYHANZQJEA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692952614 (49h17m28.045737742s) [noise=true , address=104.36.82.56:4236 , piece_id=LABJ2XYGSRS2V7K2CXDTT7R24TLXPFDZO6PK7RUK6T4LMAOE66HA, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.692993130 (49h17m28.045778248s) [noise=true , address=104.36.82.5:4846 , piece_id=35JBFYGSI2BY2G2RZOIF24HSRIHVECAVBCV4HPB3YOBFFS3LO3IQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:12MRTmrurAdfTtQqWboq2Z44yrzBKivkihfXeZ9bkCeGPcLF6iR, address:104.36.82.5:4846): protocol: rpc: dial tcp 104.36.82.5:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693003890 (49h17m28.045789018s) [noise=true , address=104.36.82.50:4013 , piece_id=CDSCV45MX3Q5BZX36X3YJIHLYMZVYE4LROBQ3TY7BNFMSD4IP4MQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: rpc: dial tcp 104.36.82.50:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693012085 (49h17m28.045797214s) [noise=true , address=104.36.82.6:4899 , piece_id=UIMN64JQNCOIIIHGNBV2LVHRTG4PGY5A54N23NM62C5RHDQ46RZQ, seg_pos={0 0}, upload=43391]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 5.161.89.252:12690->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:52.693041371 (49h17m28.045826489s) [seg_pos={0 0}, upload=43391]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Aug 31 15:25:52.711978295 (49h17m28.064763422s) [upload=43391]: Done uploading segments...
Aug 31 15:25:52.711993513 (49h17m28.064778641s) [upload=43391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 423.301367ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................X.X.......X...............] duration:
476.350117ms
FAIL (2025-08-31 15:40:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 476.370264ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................XX.XX.....................] duration:
642.290436ms
FAIL (2025-08-31 15:39:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Aug 31 15:24:43.782963718 (49h16m19.135748826s) [upload=43388]: Starting upload
Aug 31 15:24:43.783011859 (49h16m19.135796967s) [upload=43388]: Done starting upload
Aug 31 15:24:43.783050732 (49h16m19.13583583s) [upload=43388]: Uploading segments...
Aug 31 15:24:43.783221323 (49h16m19.136006431s) [upload=43388]: Got next segment. Inline: false
Aug 31 15:24:43.783253053 (49h16m19.136038161s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 31 15:24:43.783256529 (49h16m19.136041627s) [upload=43388]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 31 15:24:43.810951483 (49h16m19.163736581s) [seg_pos={0 0}, upload=43388]: Begin upload segment...
Aug 31 15:24:43.810987241 (49h16m19.163772359s) [seg_pos={0 0}, upload=43388]: Done begin upload segment.
Aug 31 15:24:43.811003081 (49h16m19.163788189s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811009112 (49h16m19.16379421s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811313755 (49h16m19.164098863s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811326890 (49h16m19.164112008s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811597299 (49h16m19.164382397s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.811934795 (49h16m19.164719893s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812258093 (49h16m19.165043182s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812522952 (49h16m19.16530805s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.812817867 (49h16m19.165602975s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813122881 (49h16m19.165907979s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813401145 (49h16m19.166186244s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813677826 (49h16m19.166462914s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.813971580 (49h16m19.166756678s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814248662 (49h16m19.167033789s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814521305 (49h16m19.167306403s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.814800902 (49h16m19.167586019s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Uploading piece...
Aug 31 15:24:43.879200864 (49h16m19.231985972s) [upload=43388]: Next returned nil segment
Aug 31 15:24:43.879225539 (49h16m19.232010647s) [upload=43388]: Waiting for error group managing segments...
Aug 31 15:24:43.888244591 (49h16m19.241029689s) [noise=true , address=45.91.116.6:4975 , piece_id=746JQRZU3MIMZJFXRZTYBIMC7JATHK7S5SU3F6JQKBKYJLXFGICA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888536120 (49h16m19.241321217s) [noise=true , address=45.91.116.4:4939 , piece_id=GAV7QOHJ6UQ2QI6VRI5MXVRKUURQL5NGVWGBADC7FZWQATTAYYIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.888758068 (49h16m19.241543166s) [noise=true , address=45.91.116.12:4225 , piece_id=A4O7ZWPRJOIUHZXMS6NHMNK4QFDGGHRETMO5NCAMNRDMWAS2BM7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889063773 (49h16m19.241848871s) [noise=true , address=77.90.31.11:4554 , piece_id=V74T364JCT5J5EMNF2MZBEJVRBHZLTRQFWN4RVYGO57FLBZX7GHQ, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.889121992 (49h16m19.241907091s) [noise=true , address=45.91.116.8:4047 , piece_id=AWFSRW5BFSZWJC4ERLN7FAC5IL4YGELUF5YXZMINTMW2Y445LEBA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.890921932 (49h16m19.24370703s) [noise=true , address=77.90.31.17:4772 , piece_id=UNTFTFPOWPUXTG5GSF2B3C5SSC2GLEABDTGFFH4ENYSWGEEGH6HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:43.895140365 (49h16m19.247925453s) [noise=true , address=77.90.31.19:4842 , piece_id=MAO3JFC4FEXHMANQRQYIQJUCC4FPDEJTSCE4LDF224AY3VXR2JRA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:24:44.374006552 (49h16m19.726791641s) [noise=true , address=77.90.31.18:4805 , piece_id=2C2I6RJLIEQJLYETHI5IQP7PP7CKJ2QWIREQIO5OZFKRX6OD4RPA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: <nil>
Aug 31 15:25:13.768479027 (49h16m49.121264156s) [noise=true , address=104.36.82.7:4909 , piece_id=NSRZ4V7XBRBRGSYFLLVZAI5MQ4K6O6D4UEE2TKDNVC56TJLC5UIA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768558637 (49h16m49.121343765s) [noise=true , address=104.36.82.9:4006 , piece_id=CESBYZLH664YM6QJ7BR5HQS5PHJ2LOSSPWVWILAWH2KIJKT2FCVA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768567263 (49h16m49.121352432s) [noise=true , address=104.36.82.3:4765 , piece_id=Q7O7TCN5FQCGJ3ITLMR2CZITUHYSZ3L6FNV73GAD2WEG7OQBA67Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768630724 (49h16m49.121415851s) [noise=true , address=104.36.82.14:4223 , piece_id=USZDSGTKKHL3LOSXDUBVSGZGNIXDLLPMCK7LNKJYHCTENHHPT6PA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768660970 (49h16m49.121446098s) [noise=true , address=104.36.82.8:4971 , piece_id=SLF4TUYKFVWG6MRV4XCSYG5WBXDQN4L7VYCW7M6L7UVMFG4H26HA, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768780635 (49h16m49.121565773s) [noise=true , address=104.36.82.5:4837 , piece_id=AVDEGAWTQWP5TYMHNVMGD7I4L2FOJU3JLS2FWPHKNWJZNSMVFJ5Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 5.161.89.252:42640->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768797638 (49h16m49.121582775s) [noise=true , address=104.36.82.16:4294 , piece_id=3QG5WSA4ZLJTYCNIZLEA23BBQPFOBY2ZNWPMWH3PHW6YYQ6YTE2Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768806394 (49h16m49.121591522s) [noise=true , address=104.36.82.2:4731 , piece_id=RH3EEVJBXRSJL5RL3MY4JQK5GKB76QS5Y5JWVRVPDI6SELALSG7Q, seg_pos={0 0}, upload=43388]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 31 15:25:13.768857169 (49h16m49.121642297s) [seg_pos={0 0}, upload=43388]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Aug 31 15:25:13.782906041 (49h16m49.135691199s) [upload=43388]: Done uploading segments...
Aug 31 15:25:13.782927661 (49h16m49.135712799s) [upload=43388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 642.308039ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration:
682.66461ms
FAIL (2025-08-31 15:39:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 682.68552ms
core
uplink
us-select-1
us1-placement-20
|