Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................................................X....XXXX] duration:
1m0.016641067s
FAIL (2025-09-03 20:37:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
eg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.807665219 (126h28m28.160450347s) [noise=true , address=104.36.82.56:4257 , piece_id=MU6GGTYTAFNLB4N6YY3SFTDWDCLHEEUY3P2RSGON4VAA4PKTFM3A, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.808125664 (126h28m28.160910793s) [noise=true , address=104.36.82.35:4612 , piece_id=TJKEU2IYCTVT4KZQVENPIVOQF6LLBMEGEXB4ZNKEBLIHIXNVSGVQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.808144079 (126h28m28.160929197s) [noise=true , address=79.127.219.47:4052 , piece_id=NH6JVWJAFBZQ2E54MIOU4QVJCF7HKOTN5YYMH6D5PWJ6I4VOCKLA, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.808503735 (126h28m28.161288864s) [noise=true , address=104.36.82.9:4019 , piece_id=W53AUZUV34XLNT5IHNYQ6EJ6ILL7QDREUCRU2JQDG7DYM6AXK2HA, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.808518483 (126h28m28.161303611s) [noise=true , address=104.36.82.61:4388 , piece_id=2KSZL4J3656GB6O3DUNLJPBKUTRM2OEC2UA4B3GVFAC2ETBENLGQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.808865996 (126h28m28.161651114s) [noise=true , address=104.36.82.58:4299 , piece_id=BZQKQW27VMEVCJD62KCEV5NLPZQGQAONHVZM6BMG5TV3BAZIVISA, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.808890322 (126h28m28.16167544s) [noise=true , address=104.36.82.23:4480 , piece_id=4YDIUNDJABZJFYY2V7ZIJM2VPE3UXVA3KA3A6BPT4OBW4THHHMYQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.809233177 (126h28m28.162018305s) [noise=true , address=104.36.82.48:4594 , piece_id=OSQJ4H27DYNR34SBYGN7J7WZAH76O4YYIGIPPKWMDFKX7CAWF3YQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.809265027 (126h28m28.162050144s) [noise=true , address=45.91.116.5:4317 , piece_id=EHY2BYYR2HZG5D7TU7N33IIUD3UGMYQUUPGMITGOOXNGBHREY6VQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.809619413 (126h28m28.162404531s) [noise=true , address=104.36.82.50:4047 , piece_id=A6D3KOFCO4FJK364UNGHTTGRSEQSTLC4P26MPAR3M2TGJNZJUMQQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.809629241 (126h28m28.16241437s) [noise=true , address=77.90.31.14:4680 , piece_id=PO3EB6YTTCK64XQIFCYHTO4MDLTYQ6CTAVDMKI4SMXA5H7X5TWGQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.810012462 (126h28m28.162797589s) [noise=true , address=104.36.82.34:4564 , piece_id=GA466AU63BKOBE2MMKBP7OCQRGJUSSTPIDYZBXCRRW5435OLLLWQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.810062735 (126h28m28.162847864s) [noise=true , address=77.90.31.15:4717 , piece_id=MZIHOOUWN77ZBVFGCYV5JWIBQUJMTTHEGU3DHMUFI7RN7ZBKVZKQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.810394029 (126h28m28.163179157s) [noise=true , address=104.36.82.54:4178 , piece_id=6QWPP6EJ4BDPOKN2EH6X2VCOO2C57T4WFKKQEE2MUTY7KWHX5SJA, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.810553499 (126h28m28.163338627s) [noise=true , address=104.36.82.55:4210 , piece_id=3QAD2MMEJX6MTFQNRZIXI3CMXEAYUP3W4RLDEVCA45KYA2CNOQAA, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.811087092 (126h28m28.163872219s) [noise=true , address=104.36.82.62:4437 , piece_id=J4HVOQ5J4LPEEIHNRA72TJW5PKU6TV77S6DPQWY5TCIPHTGCAHGQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.813296114 (126h28m28.166081242s) [noise=true , address=104.36.82.60:4355 , piece_id=V3CXQX6UWQXP4M6EX3MJKKGXNVX43KX7QDXGQC6VL5WQGISAIGQQ, seg_pos={0 0}, upload=111514]: Uploading piece...
Sep 3 20:36:52.900114148 (126h28m28.252899266s) [upload=111514]: Next returned nil segment
Sep 3 20:36:52.900127864 (126h28m28.252912982s) [upload=111514]: Waiting for error group managing segments...
Sep 3 20:36:52.932379749 (126h28m28.285164878s) [noise=true , address=77.90.31.14:4680 , piece_id=PO3EB6YTTCK64XQIFCYHTO4MDLTYQ6CTAVDMKI4SMXA5H7X5TWGQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:52.941719876 (126h28m28.294505014s) [noise=true , address=79.127.219.47:4052 , piece_id=NH6JVWJAFBZQ2E54MIOU4QVJCF7HKOTN5YYMH6D5PWJ6I4VOCKLA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:52.943004581 (126h28m28.295789709s) [noise=true , address=104.36.82.58:4299 , piece_id=BZQKQW27VMEVCJD62KCEV5NLPZQGQAONHVZM6BMG5TV3BAZIVISA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:52.948243579 (126h28m28.301028707s) [noise=true , address=77.90.31.15:4717 , piece_id=MZIHOOUWN77ZBVFGCYV5JWIBQUJMTTHEGU3DHMUFI7RN7ZBKVZKQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:52.951124274 (126h28m28.303909412s) [noise=true , address=104.36.82.61:4388 , piece_id=2KSZL4J3656GB6O3DUNLJPBKUTRM2OEC2UA4B3GVFAC2ETBENLGQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:52.967611516 (126h28m28.320396643s) [noise=true , address=45.91.116.5:4317 , piece_id=EHY2BYYR2HZG5D7TU7N33IIUD3UGMYQUUPGMITGOOXNGBHREY6VQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:52.980651166 (126h28m28.333436295s) [noise=true , address=104.36.82.62:4437 , piece_id=J4HVOQ5J4LPEEIHNRA72TJW5PKU6TV77S6DPQWY5TCIPHTGCAHGQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:52.999845226 (126h28m28.352630354s) [noise=true , address=104.36.82.54:4178 , piece_id=6QWPP6EJ4BDPOKN2EH6X2VCOO2C57T4WFKKQEE2MUTY7KWHX5SJA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:53.111093592 (126h28m28.463878731s) [noise=true , address=104.36.82.23:4480 , piece_id=4YDIUNDJABZJFYY2V7ZIJM2VPE3UXVA3KA3A6BPT4OBW4THHHMYQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:36:53.556461523 (126h28m28.909246652s) [noise=true , address=104.36.82.60:4355 , piece_id=V3CXQX6UWQXP4M6EX3MJKKGXNVX43KX7QDXGQC6VL5WQGISAIGQQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: <nil>
Sep 3 20:37:52.770263630 (126h29m28.123048738s) [noise=true , address=104.36.82.56:4257 , piece_id=MU6GGTYTAFNLB4N6YY3SFTDWDCLHEEUY3P2RSGON4VAA4PKTFM3A, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: rpc: dial tcp 104.36.82.56:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:37:52.770305489 (126h29m28.123090597s) [noise=true , address=104.36.82.48:4594 , piece_id=OSQJ4H27DYNR34SBYGN7J7WZAH76O4YYIGIPPKWMDFKX7CAWF3YQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12ACMz4vh1qJLsKN5FkjNgitVHjwmjSBJ7rF77h2NF41bQJUaPr, address:104.36.82.48:4594): protocol: rpc: dial tcp 104.36.82.48:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:37:52.770372094 (126h29m28.123157202s) [noise=true , address=104.36.82.50:4047 , piece_id=A6D3KOFCO4FJK364UNGHTTGRSEQSTLC4P26MPAR3M2TGJNZJUMQQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: write tcp 5.161.89.252:48586->104.36.82.50:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:37:52.770410878 (126h29m28.123195975s) [noise=true , address=104.36.82.51:4058 , piece_id=36TJOSLTKYEOOHHS42HABQQOSJBRJZ5R6QUFCQMZ2EC47TH5H7HQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ, address:104.36.82.51:4058): protocol: rpc: dial tcp 104.36.82.51:4058: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:37:52.770460691 (126h29m28.123245799s) [noise=true , address=104.36.82.35:4612 , piece_id=TJKEU2IYCTVT4KZQVENPIVOQF6LLBMEGEXB4ZNKEBLIHIXNVSGVQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: rpc: dial tcp 104.36.82.35:4612: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:37:52.770491839 (126h29m28.123276947s) [noise=true , address=104.36.82.9:4019 , piece_id=W53AUZUV34XLNT5IHNYQ6EJ6ILL7QDREUCRU2JQDG7DYM6AXK2HA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12JuQKLQnXiEcyCj6xthEXSPy2Wu5T1B9m4q9YteoK69rMqVydD, address:104.36.82.9:4019): protocol: rpc: dial tcp 104.36.82.9:4019: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:37:52.770500074 (126h29m28.123285183s) [noise=true , address=104.36.82.55:4210 , piece_id=3QAD2MMEJX6MTFQNRZIXI3CMXEAYUP3W4RLDEVCA45KYA2CNOQAA, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:37:52.770527787 (126h29m28.123312885s) [noise=true , address=104.36.82.34:4564 , piece_id=GA466AU63BKOBE2MMKBP7OCQRGJUSSTPIDYZBXCRRW5435OLLLWQ, seg_pos={0 0}, upload=111514]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:104.36.82.34:4564): protocol: write tcp 5.161.89.252:46828->104.36.82.34:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:37:52.770556220 (126h29m28.123341328s) [seg_pos={0 0}, upload=111514]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Sep 3 20:37:52.785614907 (126h29m28.138400015s) [upload=111514]: Done uploading segments...
Sep 3 20:37:52.785629805 (126h29m28.138414902s) [upload=111514]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.016654753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..X.X..X...................................................] duration:
363.540638ms
FAIL (2025-09-03 20:37:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 363.569352ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X...............] duration:
469.618799ms
FAIL (2025-09-03 20:37:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 469.640229ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..X.X..X..................................................] duration:
395.263799ms
FAIL (2025-09-03 20:36:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 395.289508ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................................................X....XXX] duration:
1m0.016287493s
FAIL (2025-09-03 20:36:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
6190 (126h27m28.036421288s) [upload=111501]: Uploading segments...
Sep 3 20:35:52.683758941 (126h27m28.036544049s) [upload=111501]: Got next segment. Inline: false
Sep 3 20:35:52.683783387 (126h27m28.036568495s) [upload=111501]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 20:35:52.683786914 (126h27m28.036572012s) [upload=111501]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 20:35:52.709169736 (126h27m28.061954845s) [seg_pos={0 0}, upload=111501]: Begin upload segment...
Sep 3 20:35:52.709204581 (126h27m28.061989689s) [seg_pos={0 0}, upload=111501]: Done begin upload segment.
Sep 3 20:35:52.709225231 (126h27m28.062010339s) [noise=true , address=64.49.13.154:4931 , piece_id=V3KA4HSMUHV3SRERANRFP2RV7ZJCLRWDSFUMJMMIIQ25X3QUISIA, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.709233696 (126h27m28.062018804s) [noise=true , address=45.91.116.8:4078 , piece_id=NHGJS3TR4RLUFSHOEKIDAKWSYVNXWLWC7JY6SBJCAWBDPKGSPJIQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.709553548 (126h27m28.062338656s) [noise=true , address=104.36.82.10:4097 , piece_id=J25DAURSXZENM7SQHPVP4XAOD7G7EZM3QPT6GS3GYRHPYEDSQB3Q, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.709558357 (126h27m28.062343465s) [noise=true , address=64.49.13.26:4949 , piece_id=A4CI7ZO24FTBXMDZFPKR6HVAOGLZW5VVPMQCSL5NKFP2OA4W7MRQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.709766989 (126h27m28.062552088s) [noise=true , address=77.90.31.21:4888 , piece_id=DQQ42GXJZRSYIHM4PAUMJIHOPR7N2BFYGB3YKDGIFHO5WSBUKPPA, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.709996642 (126h27m28.06278175s) [noise=true , address=104.36.82.9:4031 , piece_id=44CMWP7AVM74N3VWW3OFAFCUAII6MT5TUC6VAH4LRJNEIY6EPQTQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.710246030 (126h27m28.063031138s) [noise=true , address=64.49.13.170:4728 , piece_id=XWG3LCD4TV732ARUXBXUL3EH54L5VF2SSVLUTMD5JKEDXTIZY3RQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.710460742 (126h27m28.06324585s) [noise=true , address=104.36.82.56:4237 , piece_id=HQ6QFCRVMFHH35VXZSRJCPU62VXEJ7WUN75YJGK5WEVQXBKBO4MQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.710685114 (126h27m28.063470213s) [noise=true , address=104.36.82.52:4106 , piece_id=MJDAUMJHTLQYBSNOXWMLD7VU2EEUN6CSPGMF2VPFKD7DZ73KTYUQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.710891102 (126h27m28.06367621s) [noise=true , address=64.49.13.27:4964 , piece_id=ATR5V5PL7ENSBNQJSNTDQKYUPZUTOSSGNIC6BWEOGLFNKYAWI7SQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.711092631 (126h27m28.063877739s) [noise=true , address=104.36.82.59:4311 , piece_id=Q63FFPQLOEKMQPWPJG7WSZUQ7TE2CMCSH3DCDZGYFXLOEQ6FVHBA, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.711310370 (126h27m28.064095468s) [noise=true , address=104.36.82.12:4166 , piece_id=JT3B6SOXIJRAQBP6GQNHC5TLT5NZFYABDZFMQOV3Z7IXKAH3K62Q, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.711552696 (126h27m28.064337804s) [noise=true , address=104.36.82.62:4409 , piece_id=ON2RK3SADKYWQUYVPS3HX2U4DUWRRYDM776CYZWMV2KCGHBDK6DQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.711772569 (126h27m28.064557667s) [noise=true , address=104.36.82.45:4938 , piece_id=V42B7DQX5K3F4T5LBTVIOA7TG4OZJKY3RI7O3P4LAS37DGLVQJPA, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.712010106 (126h27m28.064795214s) [noise=true , address=104.36.82.10:4083 , piece_id=I52LHU2VBNPIDCJHNIZQ4GMW472L2QITEDEMUN4VM2XTGCLSUA4A, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.712221263 (126h27m28.06500636s) [noise=true , address=104.36.82.23:4448 , piece_id=HZKZDURCWY5X4OVJUNUWUFEV7BYAFEAZYBUTM67X67HJ55SJWATA, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.712319196 (126h27m28.065104294s) [noise=true , address=104.36.82.60:4345 , piece_id=ZAYF4BZPBNWAAGZMCSNDQBL4BZTIWDBK567CBW7CGEUFJ3YGZA5Q, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.712436537 (126h27m28.065221645s) [noise=true , address=64.49.13.140:4514 , piece_id=U4WXAU2HVGDDA4D5PXGKNGEWGZM35652RVE6CIQHXJ6PXXK6CMVQ, seg_pos={0 0}, upload=111501]: Uploading piece...
Sep 3 20:35:52.772146866 (126h27m28.124931954s) [upload=111501]: Next returned nil segment
Sep 3 20:35:52.772156284 (126h27m28.124941382s) [upload=111501]: Waiting for error group managing segments...
Sep 3 20:35:52.812129042 (126h27m28.16491414s) [noise=true , address=104.36.82.59:4311 , piece_id=Q63FFPQLOEKMQPWPJG7WSZUQ7TE2CMCSH3DCDZGYFXLOEQ6FVHBA, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:52.828801442 (126h27m28.18158654s) [noise=true , address=104.36.82.56:4237 , piece_id=HQ6QFCRVMFHH35VXZSRJCPU62VXEJ7WUN75YJGK5WEVQXBKBO4MQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:52.831707585 (126h27m28.184492672s) [noise=true , address=104.36.82.62:4409 , piece_id=ON2RK3SADKYWQUYVPS3HX2U4DUWRRYDM776CYZWMV2KCGHBDK6DQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:52.840199857 (126h27m28.192984956s) [noise=true , address=104.36.82.12:4166 , piece_id=JT3B6SOXIJRAQBP6GQNHC5TLT5NZFYABDZFMQOV3Z7IXKAH3K62Q, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:52.860429313 (126h27m28.213214411s) [noise=true , address=104.36.82.60:4345 , piece_id=ZAYF4BZPBNWAAGZMCSNDQBL4BZTIWDBK567CBW7CGEUFJ3YGZA5Q, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:52.894318797 (126h27m28.247103895s) [noise=true , address=45.91.116.8:4078 , piece_id=NHGJS3TR4RLUFSHOEKIDAKWSYVNXWLWC7JY6SBJCAWBDPKGSPJIQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:52.953801418 (126h27m28.306586526s) [noise=true , address=104.36.82.45:4938 , piece_id=V42B7DQX5K3F4T5LBTVIOA7TG4OZJKY3RI7O3P4LAS37DGLVQJPA, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:53.284385898 (126h27m28.637171026s) [noise=true , address=64.49.13.170:4728 , piece_id=XWG3LCD4TV732ARUXBXUL3EH54L5VF2SSVLUTMD5JKEDXTIZY3RQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:53.299827373 (126h27m28.652612501s) [noise=true , address=64.49.13.154:4931 , piece_id=V3KA4HSMUHV3SRERANRFP2RV7ZJCLRWDSFUMJMMIIQ25X3QUISIA, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:53.325429178 (126h27m28.678214317s) [noise=true , address=64.49.13.140:4514 , piece_id=U4WXAU2HVGDDA4D5PXGKNGEWGZM35652RVE6CIQHXJ6PXXK6CMVQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:53.340935736 (126h27m28.693720864s) [noise=true , address=64.49.13.26:4949 , piece_id=A4CI7ZO24FTBXMDZFPKR6HVAOGLZW5VVPMQCSL5NKFP2OA4W7MRQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:53.341219009 (126h27m28.694004137s) [noise=true , address=64.49.13.27:4964 , piece_id=ATR5V5PL7ENSBNQJSNTDQKYUPZUTOSSGNIC6BWEOGLFNKYAWI7SQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:35:53.361079972 (126h27m28.713865101s) [noise=true , address=104.36.82.23:4448 , piece_id=HZKZDURCWY5X4OVJUNUWUFEV7BYAFEAZYBUTM67X67HJ55SJWATA, seg_pos={0 0}, upload=111501]: Done uploading piece. err: <nil>
Sep 3 20:36:52.671336108 (126h28m28.024121205s) [noise=true , address=77.90.31.21:4888 , piece_id=DQQ42GXJZRSYIHM4PAUMJIHOPR7N2BFYGB3YKDGIFHO5WSBUKPPA, seg_pos={0 0}, upload=111501]: Done uploading piece. err: ecclient: upload failed (node:12BiqwcDuvobZabx3jxDhXf27Ei8oLFS6wLp3qbpZCtJ4ZhQEZA, address:77.90.31.21:4888): protocol: rpc: dial tcp 77.90.31.21:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:36:52.671398785 (126h28m28.024183893s) [noise=true , address=104.36.82.10:4097 , piece_id=J25DAURSXZENM7SQHPVP4XAOD7G7EZM3QPT6GS3GYRHPYEDSQB3Q, seg_pos={0 0}, upload=111501]: Done uploading piece. err: ecclient: upload failed (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3, address:104.36.82.10:4097): protocol: rpc: dial tcp 104.36.82.10:4097: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:36:52.671491900 (126h28m28.024277009s) [noise=true , address=104.36.82.52:4106 , piece_id=MJDAUMJHTLQYBSNOXWMLD7VU2EEUN6CSPGMF2VPFKD7DZ73KTYUQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: context deadline exceeded; context deadline exceeded
Sep 3 20:36:52.671504374 (126h28m28.024289482s) [noise=true , address=104.36.82.9:4031 , piece_id=44CMWP7AVM74N3VWW3OFAFCUAII6MT5TUC6VAH4LRJNEIY6EPQTQ, seg_pos={0 0}, upload=111501]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:104.36.82.9:4031): protocol: rpc: dial tcp 104.36.82.9:4031: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:36:52.671563615 (126h28m28.024348723s) [noise=true , address=104.36.82.10:4083 , piece_id=I52LHU2VBNPIDCJHNIZQ4GMW472L2QITEDEMUN4VM2XTGCLSUA4A, seg_pos={0 0}, upload=111501]: Done uploading piece. err: ecclient: upload failed (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh, address:104.36.82.10:4083): protocol: write tcp 5.161.89.252:46108->104.36.82.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:36:52.671584615 (126h28m28.024369742s) [seg_pos={0 0}, upload=111501]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 3 20:36:52.686202461 (126h28m28.03898756s) [upload=111501]: Done uploading segments...
Sep 3 20:36:52.686216037 (126h28m28.039001135s) [upload=111501]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.016299145s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [X.X..X.X..X.................................................] duration:
405.197802ms
FAIL (2025-09-03 20:36:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 405.224804ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X..............] duration:
499.668813ms
FAIL (2025-09-03 20:36:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 499.691304ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................X....XX] duration:
1m0.02389563s
FAIL (2025-09-03 20:35:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
6h26m27.950376834s) [upload=111488]: Got next segment. Inline: false
Sep 3 20:34:52.597636811 (126h26m27.950421919s) [upload=111488]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 20:34:52.597640568 (126h26m27.950425676s) [upload=111488]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 20:34:52.619337603 (126h26m27.972122711s) [seg_pos={0 0}, upload=111488]: Begin upload segment...
Sep 3 20:34:52.619364423 (126h26m27.972149521s) [seg_pos={0 0}, upload=111488]: Done begin upload segment.
Sep 3 20:34:52.619382196 (126h26m27.972167295s) [noise=true , address=104.36.82.60:4361 , piece_id=JNU6PHEV3LFVEBSKC5CDCDSYUFKJ2LM27JGOY2DICPXS2NRZVVSA, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.619426099 (126h26m27.972211197s) [noise=true , address=104.36.82.59:4310 , piece_id=PCIS2BVIKYINBMGFTJP4W6HSTXLEH64HLOWYFMWDNMCVYOR6V5CQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.619729199 (126h26m27.972514307s) [noise=true , address=64.49.13.170:4717 , piece_id=2YHVRDHHWOAYEZVOGB63JDUL6LGXLYTDL5DRL7CJZLIQGXHV6SAQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.619827874 (126h26m27.972612973s) [noise=true , address=104.36.82.12:4158 , piece_id=STAMX7MWISAYZCUWOYEAPVTVC2Z4TMEX352ZSFAZFHVG4WQ3IPUQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.620015497 (126h26m27.972800605s) [noise=true , address=104.36.82.50:4028 , piece_id=W5UAOCJXLUB2S5MNITMXEG4Z46R2TBUU35UXPKZXYBTXQCTP7HSA, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.620204783 (126h26m27.972989881s) [noise=true , address=45.91.116.14:4214 , piece_id=HZW5Q6HN4TIWDKWT3NF7DYEKMBKV2E3ZTASC2JPLKJBGJCXS6YTQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.620279664 (126h26m27.973064761s) [noise=true , address=79.127.205.246:4286 , piece_id=ISLCEFREYMQP75KQBSIBLHV3CH2JDBFCELSGCWDYNT3YBGFTPXOQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.620508714 (126h26m27.973293822s) [noise=true , address=104.36.82.55:4207 , piece_id=7U2SNOYFMPRJKGMIOJH5JCFNX3LMYICKXTEQ7M6NKSN72YMOCTHQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.620524103 (126h26m27.973309201s) [noise=true , address=77.90.31.21:4885 , piece_id=2XA64IQJAWV553O6TNPGZQBUXPUPY72GJYSH2G2BEAUTJFYNEQQQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.620781266 (126h26m27.973566374s) [noise=true , address=104.36.82.45:4926 , piece_id=VJQ5PSRP35CAUSVGPFJAZWZMZHY24D3DW6MCV3KDZTHKFNQJ3NUA, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.620787699 (126h26m27.973572796s) [noise=true , address=104.36.82.23:4473 , piece_id=UJOTTDPYLYF6RRRNLWBLJZDY4IEZPHGK3SJ6XFMYQENGP46PWUKA, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.621023913 (126h26m27.973809011s) [noise=true , address=45.91.116.9:4281 , piece_id=JOXQGKIOAQGZXKUCE5NLIEZHI7LPC7EGETJAZY2GWBX5OP4KW7WQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.621034523 (126h26m27.97381962s) [noise=true , address=64.49.13.27:4960 , piece_id=FSARHFZD5RAAA25WMVUKKZEP4UIUYVSZHYSFS5JCLSMGN2PQUGQQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.621262071 (126h26m27.974047179s) [noise=true , address=45.91.116.12:4234 , piece_id=7NGNY2S56KZPZF6PXFRWCWYHO5KFOEQFMBAPBOOU3DNLQM556OVQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.621518171 (126h26m27.97430328s) [noise=true , address=104.36.82.62:4420 , piece_id=OB4KZ4H3YJBHTM3WQMYW2EAKLTIKPK6IBD3LKBTGSARYDOKCW22A, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.622525035 (126h26m27.975310143s) [noise=true , address=104.36.82.52:4088 , piece_id=RJFE62YAZTJZ7XSDQBLWE6X5FRE7IW52BXWJEMWZ56CPKCAUGWMQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.622870163 (126h26m27.975655271s) [noise=true , address=104.36.82.40:4962 , piece_id=63OA5MX3XJOVVXJUHS3XRQQTGGC5JDWETKCWTOR2OZQIOQXPNIRQ, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.623269925 (126h26m27.976055033s) [noise=true , address=145.79.227.2:4119 , piece_id=HCPQWTCJ5CPRPFR6GKJ6BP4LYENFDZIY7OJBQPAFAKYVHYE4GJVA, seg_pos={0 0}, upload=111488]: Uploading piece...
Sep 3 20:34:52.696915338 (126h26m28.049700446s) [upload=111488]: Next returned nil segment
Sep 3 20:34:52.696933612 (126h26m28.049718711s) [upload=111488]: Waiting for error group managing segments...
Sep 3 20:34:52.725389773 (126h26m28.078174881s) [noise=true , address=145.79.227.2:4119 , piece_id=HCPQWTCJ5CPRPFR6GKJ6BP4LYENFDZIY7OJBQPAFAKYVHYE4GJVA, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:52.738968867 (126h26m28.091753976s) [noise=true , address=45.91.116.9:4281 , piece_id=JOXQGKIOAQGZXKUCE5NLIEZHI7LPC7EGETJAZY2GWBX5OP4KW7WQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:52.739588092 (126h26m28.0923732s) [noise=true , address=104.36.82.23:4473 , piece_id=UJOTTDPYLYF6RRRNLWBLJZDY4IEZPHGK3SJ6XFMYQENGP46PWUKA, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:52.744469547 (126h26m28.097254656s) [noise=true , address=104.36.82.45:4926 , piece_id=VJQ5PSRP35CAUSVGPFJAZWZMZHY24D3DW6MCV3KDZTHKFNQJ3NUA, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:52.745154485 (126h26m28.097939583s) [noise=true , address=104.36.82.59:4310 , piece_id=PCIS2BVIKYINBMGFTJP4W6HSTXLEH64HLOWYFMWDNMCVYOR6V5CQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:52.751198817 (126h26m28.103983925s) [noise=true , address=45.91.116.14:4214 , piece_id=HZW5Q6HN4TIWDKWT3NF7DYEKMBKV2E3ZTASC2JPLKJBGJCXS6YTQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:52.762166753 (126h26m28.114951851s) [noise=true , address=104.36.82.62:4420 , piece_id=OB4KZ4H3YJBHTM3WQMYW2EAKLTIKPK6IBD3LKBTGSARYDOKCW22A, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:52.836353986 (126h26m28.189139083s) [noise=true , address=45.91.116.12:4234 , piece_id=7NGNY2S56KZPZF6PXFRWCWYHO5KFOEQFMBAPBOOU3DNLQM556OVQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:52.905953794 (126h26m28.258738902s) [noise=true , address=104.36.82.60:4361 , piece_id=JNU6PHEV3LFVEBSKC5CDCDSYUFKJ2LM27JGOY2DICPXS2NRZVVSA, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:53.197927336 (126h26m28.550712444s) [noise=true , address=64.49.13.170:4717 , piece_id=2YHVRDHHWOAYEZVOGB63JDUL6LGXLYTDL5DRL7CJZLIQGXHV6SAQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:53.216682230 (126h26m28.569467358s) [noise=true , address=64.49.13.27:4960 , piece_id=FSARHFZD5RAAA25WMVUKKZEP4UIUYVSZHYSFS5JCLSMGN2PQUGQQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:53.220404017 (126h26m28.573189125s) [noise=true , address=79.127.205.246:4286 , piece_id=ISLCEFREYMQP75KQBSIBLHV3CH2JDBFCELSGCWDYNT3YBGFTPXOQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:34:53.413364970 (126h26m28.766150078s) [noise=true , address=104.36.82.12:4158 , piece_id=STAMX7MWISAYZCUWOYEAPVTVC2Z4TMEX352ZSFAZFHVG4WQ3IPUQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: <nil>
Sep 3 20:35:52.583949507 (126h27m27.936734604s) [noise=true , address=77.90.31.21:4885 , piece_id=2XA64IQJAWV553O6TNPGZQBUXPUPY72GJYSH2G2BEAUTJFYNEQQQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:35:52.583993910 (126h27m27.936779008s) [noise=true , address=104.36.82.52:4088 , piece_id=RJFE62YAZTJZ7XSDQBLWE6X5FRE7IW52BXWJEMWZ56CPKCAUGWMQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:104.36.82.52:4088): protocol: write tcp 5.161.89.252:29502->104.36.82.52:4088: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:35:52.584034997 (126h27m27.936820095s) [noise=true , address=104.36.82.40:4962 , piece_id=63OA5MX3XJOVVXJUHS3XRQQTGGC5JDWETKCWTOR2OZQIOQXPNIRQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: ecclient: upload failed (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3, address:104.36.82.40:4962): protocol: rpc: dial tcp 104.36.82.40:4962: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:35:52.584065524 (126h27m27.936850622s) [noise=true , address=104.36.82.50:4028 , piece_id=W5UAOCJXLUB2S5MNITMXEG4Z46R2TBUU35UXPKZXYBTXQCTP7HSA, seg_pos={0 0}, upload=111488]: Done uploading piece. err: ecclient: upload failed (node:12B8N15ijqmYJVmjCT74JTLcMEYSZ4xdAWxVqRRF6w8hgYJkmHF, address:104.36.82.50:4028): protocol: rpc: dial tcp 104.36.82.50:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:35:52.584076806 (126h27m27.936861904s) [noise=true , address=104.36.82.55:4207 , piece_id=7U2SNOYFMPRJKGMIOJH5JCFNX3LMYICKXTEQ7M6NKSN72YMOCTHQ, seg_pos={0 0}, upload=111488]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 5.161.89.252:29804->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:35:52.584090181 (126h27m27.93687528s) [seg_pos={0 0}, upload=111488]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 3 20:35:52.606428751 (126h27m27.959213859s) [upload=111488]: Done uploading segments...
Sep 3 20:35:52.606452195 (126h27m27.959237293s) [upload=111488]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.023912552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [XX.X..X.X..X................................................] duration:
427.611514ms
FAIL (2025-09-03 20:35:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 427.636802ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X.............] duration:
442.422222ms
FAIL (2025-09-03 20:35:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 442.443261ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.XX.X..X.X..X...............................................] duration:
380.440085ms
FAIL (2025-09-03 20:34:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 380.468648ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X....X] duration:
1m0.025919123s
FAIL (2025-09-03 20:34:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
in
Sep 3 20:33:38.182310266 (126h25m13.535095404s) [upload=111466]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 20:33:38.212779440 (126h25m13.565564538s) [seg_pos={0 0}, upload=111466]: Begin upload segment...
Sep 3 20:33:38.212822601 (126h25m13.565607709s) [seg_pos={0 0}, upload=111466]: Done begin upload segment.
Sep 3 20:33:38.212845614 (126h25m13.565630722s) [noise=true , address=104.36.82.17:4330 , piece_id=BQQ4C2NQVL3OMTQI4TSQB24CEYFKUB7UMMCYXDDD7RIHUTLBFJ4A, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.213118958 (126h25m13.565904056s) [noise=true , address=77.90.31.16:4764 , piece_id=FGPVGZRSGONCNPHZB7EUBBOA47DMEVBGYMS3ETACFYJL25GMSTHQ, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.213228404 (126h25m13.566013502s) [noise=true , address=45.91.116.5:4315 , piece_id=OCW3OC3QSXJOAUEKCVC5FQC4WQ6UGB4XWFXX7DDB6PP3SFIXDW2A, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.213481680 (126h25m13.566266778s) [noise=true , address=104.36.82.53:4156 , piece_id=P27UPROKETRCJVYJN33UZFPUN7ELY56K2VMRQZHSRMS56OLEEXUQ, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.213677508 (126h25m13.566462606s) [noise=true , address=79.127.205.245:4236 , piece_id=PSBR3SE5LDIP7CWF54OCUV2UW2YS7XAELW7MTB7Z6HUF6JO3TT4A, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.213874478 (126h25m13.566659586s) [noise=true , address=79.127.205.246:4288 , piece_id=PXFEFPGFZZASIJQCSFBBRAD6QEF3UUAXND62EIBOFKRMGYJZVO4Q, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.213977782 (126h25m13.566762881s) [noise=true , address=104.36.82.10:4103 , piece_id=ARGBZDPQGNDBTUGG7BGNWTEHOICS3C3LRIPPLYFPBJMIAYJSKKLA, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.214136802 (126h25m13.56692191s) [noise=true , address=77.90.31.12:4602 , piece_id=JSC5TIXHPHR4RGOEII7BFUDZ7WKRPPQKA526PA7CYMJFE3ZIEIZQ, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.214249533 (126h25m13.567034641s) [noise=true , address=104.36.82.54:4159 , piece_id=TL4TBJS66C22DTJJP34C2VKZKYESWFE5VKOFCM7DLHDUMSFYSJBQ, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.214455140 (126h25m13.567240248s) [noise=true , address=77.90.31.14:4677 , piece_id=KBLYDHSZIUHM3E6LVX2PW2QJXICHKXWAGFORAD6PK4FEBVAZMXMA, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.214692797 (126h25m13.567477895s) [noise=true , address=104.36.82.3:4771 , piece_id=Y2LVRTWRXXW5H7NNLLPOVKZVE34GFDSZYV3FHBYVGZ5T7BKYRSFQ, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.214789778 (126h25m13.567574887s) [noise=true , address=104.36.82.50:4046 , piece_id=HRQVNCABXWBWF2PGNYNQ4GJKRDSTBNJGNET7OM2QAGNGOEWWZFHQ, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.215048296 (126h25m13.567833404s) [noise=true , address=104.36.82.59:4335 , piece_id=VS6KPSZDFAEPZYQZS2LVSVYSV4GDH2WN3PBPPRZHDME4477KFRQA, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.215125099 (126h25m13.567910207s) [noise=true , address=104.36.82.8:4991 , piece_id=FZ5XCYD4S4G2656XMRJW2SX2SXFUJDIEUEOHSNPAOKZFMTLKYKZA, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.215369218 (126h25m13.568154326s) [noise=true , address=104.36.82.51:4053 , piece_id=7S42EQRMQKFBG7FCF4T4ISSEL6YGTNZCFJGKCZRTZ4SIPI4KZGDA, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.215825165 (126h25m13.568610273s) [noise=true , address=104.36.82.60:4340 , piece_id=WQMP6Z4RJ4MBCSDAAQ2DIUN5BWKBWFUVNFKKGWDQ55O3YK3NZJ4A, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.216226440 (126h25m13.569011588s) [noise=true , address=45.91.116.4:4941 , piece_id=OORMZZUFYP5C2XL2E2KQXG3OGDYFA7A4L6COXDERFL2L632D46WA, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.217272746 (126h25m13.570057874s) [noise=true , address=104.36.82.23:4473 , piece_id=4F2QNQLYMWIXI6RQSZALIEYMZU6ZPNYVN7ROY4FONRJDZWVO3TVA, seg_pos={0 0}, upload=111466]: Uploading piece...
Sep 3 20:33:38.308440476 (126h25m13.661225574s) [upload=111466]: Next returned nil segment
Sep 3 20:33:38.308451968 (126h25m13.661237066s) [upload=111466]: Waiting for error group managing segments...
Sep 3 20:33:38.339522102 (126h25m13.6923072s) [noise=true , address=104.36.82.59:4335 , piece_id=VS6KPSZDFAEPZYQZS2LVSVYSV4GDH2WN3PBPPRZHDME4477KFRQA, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.340242486 (126h25m13.693027584s) [noise=true , address=45.91.116.4:4941 , piece_id=OORMZZUFYP5C2XL2E2KQXG3OGDYFA7A4L6COXDERFL2L632D46WA, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.351036566 (126h25m13.703821664s) [noise=true , address=77.90.31.14:4677 , piece_id=KBLYDHSZIUHM3E6LVX2PW2QJXICHKXWAGFORAD6PK4FEBVAZMXMA, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.369062851 (126h25m13.721847949s) [noise=true , address=77.90.31.16:4764 , piece_id=FGPVGZRSGONCNPHZB7EUBBOA47DMEVBGYMS3ETACFYJL25GMSTHQ, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.382310683 (126h25m13.735095801s) [noise=true , address=104.36.82.17:4330 , piece_id=BQQ4C2NQVL3OMTQI4TSQB24CEYFKUB7UMMCYXDDD7RIHUTLBFJ4A, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.499699282 (126h25m13.85248438s) [noise=true , address=104.36.82.23:4473 , piece_id=4F2QNQLYMWIXI6RQSZALIEYMZU6ZPNYVN7ROY4FONRJDZWVO3TVA, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.502436095 (126h25m13.855221203s) [noise=true , address=104.36.82.54:4159 , piece_id=TL4TBJS66C22DTJJP34C2VKZKYESWFE5VKOFCM7DLHDUMSFYSJBQ, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.635315704 (126h25m13.988100802s) [noise=true , address=45.91.116.5:4315 , piece_id=OCW3OC3QSXJOAUEKCVC5FQC4WQ6UGB4XWFXX7DDB6PP3SFIXDW2A, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.737471469 (126h25m14.090256607s) [noise=true , address=77.90.31.12:4602 , piece_id=JSC5TIXHPHR4RGOEII7BFUDZ7WKRPPQKA526PA7CYMJFE3ZIEIZQ, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.756336671 (126h25m14.109121809s) [noise=true , address=79.127.205.245:4236 , piece_id=PSBR3SE5LDIP7CWF54OCUV2UW2YS7XAELW7MTB7Z6HUF6JO3TT4A, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.765191015 (126h25m14.117976143s) [noise=true , address=79.127.205.246:4288 , piece_id=PXFEFPGFZZASIJQCSFBBRAD6QEF3UUAXND62EIBOFKRMGYJZVO4Q, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:33:38.835334987 (126h25m14.188120124s) [noise=true , address=104.36.82.60:4340 , piece_id=WQMP6Z4RJ4MBCSDAAQ2DIUN5BWKBWFUVNFKKGWDQ55O3YK3NZJ4A, seg_pos={0 0}, upload=111466]: Done uploading piece. err: <nil>
Sep 3 20:34:38.164899720 (126h26m13.517684828s) [noise=true , address=104.36.82.50:4046 , piece_id=HRQVNCABXWBWF2PGNYNQ4GJKRDSTBNJGNET7OM2QAGNGOEWWZFHQ, seg_pos={0 0}, upload=111466]: Done uploading piece. err: ecclient: upload failed (node:12EsoehnMdbRs4cRP66HD93ZWZuzpMWYkfo1VxvQLyQcodAEm7C, address:104.36.82.50:4046): protocol: rpc: dial tcp 104.36.82.50:4046: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:34:38.164948532 (126h26m13.51773364s) [noise=true , address=104.36.82.51:4053 , piece_id=7S42EQRMQKFBG7FCF4T4ISSEL6YGTNZCFJGKCZRTZ4SIPI4KZGDA, seg_pos={0 0}, upload=111466]: Done uploading piece. err: ecclient: upload failed (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG, address:104.36.82.51:4053): protocol: write tcp 5.161.89.252:26794->104.36.82.51:4053: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:34:38.164969272 (126h26m13.51775438s) [noise=true , address=104.36.82.3:4771 , piece_id=Y2LVRTWRXXW5H7NNLLPOVKZVE34GFDSZYV3FHBYVGZ5T7BKYRSFQ, seg_pos={0 0}, upload=111466]: Done uploading piece. err: ecclient: upload failed (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL, address:104.36.82.3:4771): protocol: rpc: dial tcp 104.36.82.3:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:34:38.164985512 (126h26m13.517770611s) [noise=true , address=104.36.82.53:4156 , piece_id=P27UPROKETRCJVYJN33UZFPUN7ELY56K2VMRQZHSRMS56OLEEXUQ, seg_pos={0 0}, upload=111466]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: rpc: dial tcp 104.36.82.53:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:34:38.165007464 (126h26m13.517792572s) [noise=true , address=104.36.82.10:4103 , piece_id=ARGBZDPQGNDBTUGG7BGNWTEHOICS3C3LRIPPLYFPBJMIAYJSKKLA, seg_pos={0 0}, upload=111466]: Done uploading piece. err: ecclient: upload failed (node:125eQFBenzeVGCxBzzDUQDNydyAntAWGWcq38CXqjmVt3qG5Qbn, address:104.36.82.10:4103): protocol: rpc: dial tcp 104.36.82.10:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:34:38.165020258 (126h26m13.517805356s) [noise=true , address=104.36.82.8:4991 , piece_id=FZ5XCYD4S4G2656XMRJW2SX2SXFUJDIEUEOHSNPAOKZFMTLKYKZA, seg_pos={0 0}, upload=111466]: Done uploading piece. err: ecclient: upload failed (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx, address:104.36.82.8:4991): protocol: rpc: dial tcp 104.36.82.8:4991: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:34:38.165040316 (126h26m13.517825424s) [seg_pos={0 0}, upload=111466]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 3 20:34:38.189266113 (126h26m13.542051211s) [upload=111466]: Done uploading segments...
Sep 3 20:34:38.189288084 (126h26m13.542073182s) [upload=111466]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.025951333s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..XX.X..X.X..X..............................................] duration:
7.511335825s
FAIL (2025-09-03 20:34:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 7.511366142s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X............] duration:
1.199762279s
FAIL (2025-09-03 20:34:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.199784081s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X...........] duration:
451.621138ms
FAIL (2025-09-03 20:33:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 451.639261ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...XX.X..X.X..X.............................................] duration:
418.680617ms
FAIL (2025-09-03 20:32:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 418.704252ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
1.094517799s
FAIL (2025-09-03 20:32:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
5207043s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 20:29:05.122432023 (126h20m40.475217151s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 20:29:05.146531426 (126h20m40.499316614s) [seg_pos={0 0}, upload=111402]: Begin upload segment...
Sep 3 20:29:05.146605506 (126h20m40.499390633s) [seg_pos={0 0}, upload=111402]: Done begin upload segment.
Sep 3 20:29:05.146684304 (126h20m40.499469433s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.146707899 (126h20m40.499493037s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147247083 (126h20m40.500032221s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147272099 (126h20m40.500057238s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147694604 (126h20m40.500479732s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147788049 (126h20m40.500573177s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148112731 (126h20m40.500897858s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148314189 (126h20m40.501099317s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148501101 (126h20m40.501286229s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148843695 (126h20m40.501628823s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148894360 (126h20m40.501679488s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149245851 (126h20m40.502030989s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149370034 (126h20m40.502155173s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149597523 (126h20m40.502382651s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150079800 (126h20m40.502864927s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150476385 (126h20m40.503261524s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150870757 (126h20m40.503655895s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.151291468 (126h20m40.504076596s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.261085644 (126h20m40.613870772s) [upload=111402]: Next returned nil segment
Sep 3 20:29:05.261106263 (126h20m40.613891392s) [upload=111402]: Waiting for error group managing segments...
Sep 3 20:29:05.282548659 (126h20m40.635333757s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296015366 (126h20m40.648800463s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296416961 (126h20m40.649202069s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.297306704 (126h20m40.650091812s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.305865911 (126h20m40.65865104s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.310225140 (126h20m40.663010248s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.331485182 (126h20m40.68427027s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.343081982 (126h20m40.69586708s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.344200816 (126h20m40.696985914s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.530920283 (126h20m40.883705381s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.703184575 (126h20m41.055969703s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.905385085 (126h20m41.258170193s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:06.047360604 (126h20m41.400145732s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:30:05.108299836 (126h21m40.461084934s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC, address:104.36.82.8:4979): protocol: write tcp 5.161.89.252:30880->104.36.82.8:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108350412 (126h21m40.46113551s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:1HZ945gjHWqLgS2Zccf7NTZCY7RLdnPLUebH2ZPEytMuyjmvvF, address:104.36.82.8:4986): protocol: write tcp 5.161.89.252:46632->104.36.82.8:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108510673 (126h21m40.461295781s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: write tcp 5.161.89.252:38864->104.36.82.42:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108522214 (126h21m40.461307352s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:104.36.82.42:4832): protocol: write tcp 5.161.89.252:43688->104.36.82.42:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108548554 (126h21m40.461333663s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12gTRhEnzLoKmQYRttr4hwiTbJ4y3yqMUS1Qg7F2yohb81sd5KV, address:104.36.82.4:4812): protocol: write tcp 5.161.89.252:37676->104.36.82.4:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108565316 (126h21m40.461350424s) [seg_pos={0 0}, upload=111402]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 3 20:30:05.122647449 (126h21m40.475432577s) [upload=111402]: Done uploading segments...
Sep 3 20:30:05.122679280 (126h21m40.475464408s) [upload=111402]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.094535342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X..........] duration:
642.409718ms
FAIL (2025-09-03 20:32:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 642.429344ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....XX.X..X.X..X............................................] duration:
528.942823ms
FAIL (2025-09-03 20:32:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 528.961548ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [....X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X.........] duration:
765.082595ms
FAIL (2025-09-03 20:31:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 765.104043ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....XX.X..X.X..X...........................................] duration:
549.534401ms
FAIL (2025-09-03 20:31:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 549.58199ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
3.34282638s
FAIL (2025-09-03 20:31:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
5207043s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 20:29:05.122432023 (126h20m40.475217151s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 20:29:05.146531426 (126h20m40.499316614s) [seg_pos={0 0}, upload=111402]: Begin upload segment...
Sep 3 20:29:05.146605506 (126h20m40.499390633s) [seg_pos={0 0}, upload=111402]: Done begin upload segment.
Sep 3 20:29:05.146684304 (126h20m40.499469433s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.146707899 (126h20m40.499493037s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147247083 (126h20m40.500032221s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147272099 (126h20m40.500057238s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147694604 (126h20m40.500479732s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147788049 (126h20m40.500573177s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148112731 (126h20m40.500897858s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148314189 (126h20m40.501099317s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148501101 (126h20m40.501286229s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148843695 (126h20m40.501628823s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148894360 (126h20m40.501679488s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149245851 (126h20m40.502030989s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149370034 (126h20m40.502155173s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149597523 (126h20m40.502382651s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150079800 (126h20m40.502864927s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150476385 (126h20m40.503261524s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150870757 (126h20m40.503655895s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.151291468 (126h20m40.504076596s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.261085644 (126h20m40.613870772s) [upload=111402]: Next returned nil segment
Sep 3 20:29:05.261106263 (126h20m40.613891392s) [upload=111402]: Waiting for error group managing segments...
Sep 3 20:29:05.282548659 (126h20m40.635333757s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296015366 (126h20m40.648800463s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296416961 (126h20m40.649202069s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.297306704 (126h20m40.650091812s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.305865911 (126h20m40.65865104s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.310225140 (126h20m40.663010248s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.331485182 (126h20m40.68427027s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.343081982 (126h20m40.69586708s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.344200816 (126h20m40.696985914s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.530920283 (126h20m40.883705381s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.703184575 (126h20m41.055969703s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.905385085 (126h20m41.258170193s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:06.047360604 (126h20m41.400145732s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:30:05.108299836 (126h21m40.461084934s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC, address:104.36.82.8:4979): protocol: write tcp 5.161.89.252:30880->104.36.82.8:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108350412 (126h21m40.46113551s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:1HZ945gjHWqLgS2Zccf7NTZCY7RLdnPLUebH2ZPEytMuyjmvvF, address:104.36.82.8:4986): protocol: write tcp 5.161.89.252:46632->104.36.82.8:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108510673 (126h21m40.461295781s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: write tcp 5.161.89.252:38864->104.36.82.42:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108522214 (126h21m40.461307352s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:104.36.82.42:4832): protocol: write tcp 5.161.89.252:43688->104.36.82.42:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108548554 (126h21m40.461333663s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12gTRhEnzLoKmQYRttr4hwiTbJ4y3yqMUS1Qg7F2yohb81sd5KV, address:104.36.82.4:4812): protocol: write tcp 5.161.89.252:37676->104.36.82.4:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108565316 (126h21m40.461350424s) [seg_pos={0 0}, upload=111402]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 3 20:30:05.122647449 (126h21m40.475432577s) [upload=111402]: Done uploading segments...
Sep 3 20:30:05.122679280 (126h21m40.475464408s) [upload=111402]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.342850445s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.....X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X........] duration:
1.851304534s
FAIL (2025-09-03 20:31:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.851326215s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
1.13397074s
FAIL (2025-09-03 20:30:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
5207043s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 20:29:05.122432023 (126h20m40.475217151s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 20:29:05.146531426 (126h20m40.499316614s) [seg_pos={0 0}, upload=111402]: Begin upload segment...
Sep 3 20:29:05.146605506 (126h20m40.499390633s) [seg_pos={0 0}, upload=111402]: Done begin upload segment.
Sep 3 20:29:05.146684304 (126h20m40.499469433s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.146707899 (126h20m40.499493037s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147247083 (126h20m40.500032221s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147272099 (126h20m40.500057238s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147694604 (126h20m40.500479732s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147788049 (126h20m40.500573177s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148112731 (126h20m40.500897858s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148314189 (126h20m40.501099317s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148501101 (126h20m40.501286229s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148843695 (126h20m40.501628823s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148894360 (126h20m40.501679488s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149245851 (126h20m40.502030989s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149370034 (126h20m40.502155173s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149597523 (126h20m40.502382651s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150079800 (126h20m40.502864927s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150476385 (126h20m40.503261524s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150870757 (126h20m40.503655895s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.151291468 (126h20m40.504076596s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.261085644 (126h20m40.613870772s) [upload=111402]: Next returned nil segment
Sep 3 20:29:05.261106263 (126h20m40.613891392s) [upload=111402]: Waiting for error group managing segments...
Sep 3 20:29:05.282548659 (126h20m40.635333757s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296015366 (126h20m40.648800463s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296416961 (126h20m40.649202069s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.297306704 (126h20m40.650091812s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.305865911 (126h20m40.65865104s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.310225140 (126h20m40.663010248s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.331485182 (126h20m40.68427027s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.343081982 (126h20m40.69586708s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.344200816 (126h20m40.696985914s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.530920283 (126h20m40.883705381s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.703184575 (126h20m41.055969703s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.905385085 (126h20m41.258170193s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:06.047360604 (126h20m41.400145732s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:30:05.108299836 (126h21m40.461084934s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC, address:104.36.82.8:4979): protocol: write tcp 5.161.89.252:30880->104.36.82.8:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108350412 (126h21m40.46113551s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:1HZ945gjHWqLgS2Zccf7NTZCY7RLdnPLUebH2ZPEytMuyjmvvF, address:104.36.82.8:4986): protocol: write tcp 5.161.89.252:46632->104.36.82.8:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108510673 (126h21m40.461295781s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: write tcp 5.161.89.252:38864->104.36.82.42:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108522214 (126h21m40.461307352s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:104.36.82.42:4832): protocol: write tcp 5.161.89.252:43688->104.36.82.42:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108548554 (126h21m40.461333663s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12gTRhEnzLoKmQYRttr4hwiTbJ4y3yqMUS1Qg7F2yohb81sd5KV, address:104.36.82.4:4812): protocol: write tcp 5.161.89.252:37676->104.36.82.4:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108565316 (126h21m40.461350424s) [seg_pos={0 0}, upload=111402]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 3 20:30:05.122647449 (126h21m40.475432577s) [upload=111402]: Done uploading segments...
Sep 3 20:30:05.122679280 (126h21m40.475464408s) [upload=111402]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.133994393s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [......X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X.......] duration:
709.939217ms
FAIL (2025-09-03 20:30:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 709.958734ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......XX.X..X.X..X..........................................] duration:
474.009573ms
FAIL (2025-09-03 20:30:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 474.185234ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.339945727s
FAIL (2025-09-03 20:30:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
5207043s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 20:29:05.122432023 (126h20m40.475217151s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 20:29:05.146531426 (126h20m40.499316614s) [seg_pos={0 0}, upload=111402]: Begin upload segment...
Sep 3 20:29:05.146605506 (126h20m40.499390633s) [seg_pos={0 0}, upload=111402]: Done begin upload segment.
Sep 3 20:29:05.146684304 (126h20m40.499469433s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.146707899 (126h20m40.499493037s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147247083 (126h20m40.500032221s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147272099 (126h20m40.500057238s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147694604 (126h20m40.500479732s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147788049 (126h20m40.500573177s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148112731 (126h20m40.500897858s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148314189 (126h20m40.501099317s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148501101 (126h20m40.501286229s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148843695 (126h20m40.501628823s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148894360 (126h20m40.501679488s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149245851 (126h20m40.502030989s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149370034 (126h20m40.502155173s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149597523 (126h20m40.502382651s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150079800 (126h20m40.502864927s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150476385 (126h20m40.503261524s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150870757 (126h20m40.503655895s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.151291468 (126h20m40.504076596s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.261085644 (126h20m40.613870772s) [upload=111402]: Next returned nil segment
Sep 3 20:29:05.261106263 (126h20m40.613891392s) [upload=111402]: Waiting for error group managing segments...
Sep 3 20:29:05.282548659 (126h20m40.635333757s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296015366 (126h20m40.648800463s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296416961 (126h20m40.649202069s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.297306704 (126h20m40.650091812s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.305865911 (126h20m40.65865104s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.310225140 (126h20m40.663010248s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.331485182 (126h20m40.68427027s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.343081982 (126h20m40.69586708s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.344200816 (126h20m40.696985914s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.530920283 (126h20m40.883705381s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.703184575 (126h20m41.055969703s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.905385085 (126h20m41.258170193s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:06.047360604 (126h20m41.400145732s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:30:05.108299836 (126h21m40.461084934s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC, address:104.36.82.8:4979): protocol: write tcp 5.161.89.252:30880->104.36.82.8:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108350412 (126h21m40.46113551s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:1HZ945gjHWqLgS2Zccf7NTZCY7RLdnPLUebH2ZPEytMuyjmvvF, address:104.36.82.8:4986): protocol: write tcp 5.161.89.252:46632->104.36.82.8:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108510673 (126h21m40.461295781s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: write tcp 5.161.89.252:38864->104.36.82.42:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108522214 (126h21m40.461307352s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:104.36.82.42:4832): protocol: write tcp 5.161.89.252:43688->104.36.82.42:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108548554 (126h21m40.461333663s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12gTRhEnzLoKmQYRttr4hwiTbJ4y3yqMUS1Qg7F2yohb81sd5KV, address:104.36.82.4:4812): protocol: write tcp 5.161.89.252:37676->104.36.82.4:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108565316 (126h21m40.461350424s) [seg_pos={0 0}, upload=111402]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 3 20:30:05.122647449 (126h21m40.475432577s) [upload=111402]: Done uploading segments...
Sep 3 20:30:05.122679280 (126h21m40.475464408s) [upload=111402]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.339967298s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.017654954s
FAIL (2025-09-03 20:30:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
5207043s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 20:29:05.122432023 (126h20m40.475217151s) [upload=111402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 20:29:05.146531426 (126h20m40.499316614s) [seg_pos={0 0}, upload=111402]: Begin upload segment...
Sep 3 20:29:05.146605506 (126h20m40.499390633s) [seg_pos={0 0}, upload=111402]: Done begin upload segment.
Sep 3 20:29:05.146684304 (126h20m40.499469433s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.146707899 (126h20m40.499493037s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147247083 (126h20m40.500032221s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147272099 (126h20m40.500057238s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147694604 (126h20m40.500479732s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.147788049 (126h20m40.500573177s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148112731 (126h20m40.500897858s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148314189 (126h20m40.501099317s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148501101 (126h20m40.501286229s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148843695 (126h20m40.501628823s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.148894360 (126h20m40.501679488s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149245851 (126h20m40.502030989s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149370034 (126h20m40.502155173s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.149597523 (126h20m40.502382651s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150079800 (126h20m40.502864927s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150476385 (126h20m40.503261524s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.150870757 (126h20m40.503655895s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.151291468 (126h20m40.504076596s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Uploading piece...
Sep 3 20:29:05.261085644 (126h20m40.613870772s) [upload=111402]: Next returned nil segment
Sep 3 20:29:05.261106263 (126h20m40.613891392s) [upload=111402]: Waiting for error group managing segments...
Sep 3 20:29:05.282548659 (126h20m40.635333757s) [noise=true , address=79.127.219.51:4194 , piece_id=CS3JLT6PKDS3X6RAX5ECUIL7OKSA6IKR4X42IP24SXEWVLWOY4XA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296015366 (126h20m40.648800463s) [noise=true , address=104.36.82.55:4207 , piece_id=PW5NQ7EXIA45YM64ZBVCSE6ETCJ4LXQAXPWOOD4DQ3CNV3P6P37Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.296416961 (126h20m40.649202069s) [noise=true , address=104.36.82.58:4298 , piece_id=ETSNNE2KA4JDLUOB4PEJSVOS2UUFQRDO2F54NY2TIHPCOZ2LBKYQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.297306704 (126h20m40.650091812s) [noise=true , address=45.91.116.9:4291 , piece_id=4GAKZPJTTKWYWM26SXTHGXPVEPHFM6DKCTJVZSQF4YWEUQKXQUZQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.305865911 (126h20m40.65865104s) [noise=true , address=104.36.82.60:4363 , piece_id=557ZQEPRSYNBCGMGP6UL5IUBZHOS4X3CQ2RLIBNRMBQVHH7Q4GRQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.310225140 (126h20m40.663010248s) [noise=true , address=77.90.31.12:4613 , piece_id=JMUXCLUTBRLCDGQRADLEDSROM4AU6L6S2COADEWZDOKGTPN4CKPA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.331485182 (126h20m40.68427027s) [noise=true , address=77.90.31.10:4530 , piece_id=BG56HMDXP2XJAIM5AHYRCU2NN3XGCKOTKNARUHC5YINC7I66VSNQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.343081982 (126h20m40.69586708s) [noise=true , address=77.90.31.19:4866 , piece_id=I44SW3KSJKY62XGNHLZ2DUF3BPZS747IWHI7Q73JXAJOOBNNDW2Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.344200816 (126h20m40.696985914s) [noise=true , address=45.91.116.5:4309 , piece_id=CQ7NHPV4Z4U6X2JEG7OIWSGUBZDPJQSMCTL3BP7YCKUOIOORB54Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.530920283 (126h20m40.883705381s) [noise=true , address=45.91.116.13:4170 , piece_id=HRV3XTP7PKHEJTP4SHDMHAVEDJTUQKNHUWLTV7FDV3YQSTJRPNFQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.703184575 (126h20m41.055969703s) [noise=true , address=79.127.205.246:4266 , piece_id=WI2YO3FWFCOCJR4OJKJRE75GRTTNWOGA6OJKSF55MWOO3MOS2SSQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:05.905385085 (126h20m41.258170193s) [noise=true , address=104.36.82.12:4154 , piece_id=P4CDRQRG6JGRVAPWU65PO5WKNYHIGXH6EHIDKX6NJJTIFQKWDZMQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:29:06.047360604 (126h20m41.400145732s) [noise=true , address=104.36.82.23:4478 , piece_id=5BM4QUXI5W4BPOM3Q3TJBYQREGFZBFABG4M76MP5U3EDUZ3VKGSA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: <nil>
Sep 3 20:30:05.108299836 (126h21m40.461084934s) [noise=true , address=104.36.82.8:4979 , piece_id=3XBE6PYKN2MVBVZ4SMFFO34632I5WHK4J27MT532D4CJT3V2EL5Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC, address:104.36.82.8:4979): protocol: write tcp 5.161.89.252:30880->104.36.82.8:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108350412 (126h21m40.46113551s) [noise=true , address=104.36.82.8:4986 , piece_id=TU5A37MNAS52UYPJGZVGEQH4RUUXALHLLV3LWHJBP5MGV56WEJ7A, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:1HZ945gjHWqLgS2Zccf7NTZCY7RLdnPLUebH2ZPEytMuyjmvvF, address:104.36.82.8:4986): protocol: write tcp 5.161.89.252:46632->104.36.82.8:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108510673 (126h21m40.461295781s) [noise=true , address=104.36.82.42:4831 , piece_id=RDFRWXYNMFR4SFELR32MFUSIEUFXNUFDL55DT7JKQWMXJFBZNJWQ, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: write tcp 5.161.89.252:38864->104.36.82.42:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108522214 (126h21m40.461307352s) [noise=true , address=104.36.82.42:4832 , piece_id=5QP7GPYG6Q67MOQX5KU4HW3RRMLOV2RWEKHU3SNRF2OFFXS7OS4Q, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:104.36.82.42:4832): protocol: write tcp 5.161.89.252:43688->104.36.82.42:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108548554 (126h21m40.461333663s) [noise=true , address=104.36.82.4:4812 , piece_id=IEOW656TKM4TTTHQ4XHY4LQFRWR465RTZVL64FRN7ZFQPIBAJ4LA, seg_pos={0 0}, upload=111402]: Done uploading piece. err: ecclient: upload failed (node:12gTRhEnzLoKmQYRttr4hwiTbJ4y3yqMUS1Qg7F2yohb81sd5KV, address:104.36.82.4:4812): protocol: write tcp 5.161.89.252:37676->104.36.82.4:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 20:30:05.108565316 (126h21m40.461350424s) [seg_pos={0 0}, upload=111402]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 3 20:30:05.122647449 (126h21m40.475432577s) [upload=111402]: Done uploading segments...
Sep 3 20:30:05.122679280 (126h21m40.475464408s) [upload=111402]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.017676905s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.......X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X......] duration:
403.620573ms
FAIL (2025-09-03 20:29:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 403.644749ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......XX.X..X.X..X.........................................] duration:
425.829413ms
FAIL (2025-09-03 20:29:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 425.850241ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [........X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X.....] duration:
440.69069ms
FAIL (2025-09-03 20:29:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 440.721457ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........XX.X..X.X..X........................................] duration:
622.947624ms
FAIL (2025-09-03 20:29:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 622.965639ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........XX.X..X.X..X.......................................] duration:
606.371892ms
FAIL (2025-09-03 20:28:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 606.407799ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.........X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X....] duration:
415.467035ms
FAIL (2025-09-03 20:28:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 415.486722ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........XX.X..X.X..X......................................] duration:
454.658941ms
FAIL (2025-09-03 20:27:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 454.67934ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..........X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X...] duration:
409.734433ms
FAIL (2025-09-03 20:27:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 409.768186ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........XX.X..X.X..X.....................................] duration:
408.388249ms
FAIL (2025-09-03 20:27:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 408.411503ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...........X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X..] duration:
437.129304ms
FAIL (2025-09-03 20:27:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 437.196561ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............XX.X..X.X..X....................................] duration:
407.175019ms
FAIL (2025-09-03 20:26:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 407.195398ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [............X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X.] duration:
600.437251ms
FAIL (2025-09-03 20:25:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 600.486515ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.............X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....X] duration:
30.00078628s
FAIL (2025-09-03 20:25:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000834692s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............XX.X..X.X..X...................................] duration:
408.826443ms
FAIL (2025-09-03 20:25:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 408.853455ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..............X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.....] duration:
426.322961ms
FAIL (2025-09-03 20:24:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 426.342808ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............XX.X..X.X..X..................................] duration:
663.552791ms
FAIL (2025-09-03 20:24:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 663.575384ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [...............X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX....] duration:
493.174915ms
FAIL (2025-09-03 20:23:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 493.200213ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............XX.X..X.X..X.................................] duration:
356.069898ms
FAIL (2025-09-03 20:23:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 356.092501ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [................X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX...] duration:
424.823369ms
FAIL (2025-09-03 20:22:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 424.843849ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.................X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX..] duration:
460.943611ms
FAIL (2025-09-03 20:22:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 461.060159ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [................XX.X..X.X..X................................] duration:
471.005865ms
FAIL (2025-09-03 20:22:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
ep 3 19:56:45.732224789 (125h48m21.085009917s) [upload=110928]: Done starting upload
Sep 3 19:56:45.732366416 (125h48m21.085151584s) [upload=110928]: Uploading segments...
Sep 3 19:56:45.732644508 (125h48m21.085429637s) [upload=110928]: Got next segment. Inline: false
Sep 3 19:56:45.732696506 (125h48m21.085481644s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 3 19:56:45.732702056 (125h48m21.085487185s) [upload=110928]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 3 19:56:45.752727619 (125h48m21.105512746s) [seg_pos={0 0}, upload=110928]: Begin upload segment...
Sep 3 19:56:45.752766493 (125h48m21.10555161s) [seg_pos={0 0}, upload=110928]: Done begin upload segment.
Sep 3 19:56:45.752792240 (125h48m21.105577349s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.752815184 (125h48m21.105600312s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753170212 (125h48m21.10595533s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753336484 (125h48m21.106121611s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.753503347 (125h48m21.106288455s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.758511346 (125h48m21.111296474s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759016185 (125h48m21.111801314s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.759578112 (125h48m21.112363239s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.760792807 (125h48m21.113577935s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762064117 (125h48m21.114849255s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762554089 (125h48m21.115339217s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.762985740 (125h48m21.115770869s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.763702078 (125h48m21.116487206s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.764428504 (125h48m21.117213631s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.765629592 (125h48m21.11841472s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.769552200 (125h48m21.122337328s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Uploading piece...
Sep 3 19:56:45.886050033 (125h48m21.238835161s) [upload=110928]: Next returned nil segment
Sep 3 19:56:45.886067846 (125h48m21.238852974s) [upload=110928]: Waiting for error group managing segments...
Sep 3 19:56:45.904065404 (125h48m21.256850521s) [noise=true , address=77.90.31.15:4698 , piece_id=N5TQBVF6QRI2ENECJY7DJWHNA3M66MIEGXFDWI57UOFA7GPAQORQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.908807473 (125h48m21.2615926s) [noise=true , address=45.91.116.11:4117 , piece_id=5EF5662CB3JEYOQUSIRIO3OS5FZH657ZUHPDNUTJFPCZCQQVPEIA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.909302953 (125h48m21.262088072s) [noise=true , address=45.91.116.14:4190 , piece_id=TSBHZS7PK6RO2DEBFUR6HZFDS2QOMPXDP2V3WEPU5QRU2QE7NXTA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.915657063 (125h48m21.268442192s) [noise=true , address=45.91.116.6:4975 , piece_id=LLKBHDI65LUJQMLH72CVHIKOAH3CZTFD4CVWWZDNM4M754OTTMWQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.935856251 (125h48m21.288641349s) [noise=true , address=77.90.31.21:4877 , piece_id=7NSHHBXL26GVBSSU5B43NH4YP6J3H4AQUZNQNDSPFA6YR2VBJ72Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:45.938858257 (125h48m21.291643365s) [noise=true , address=45.91.116.12:4226 , piece_id=CPN6M2MLVZR4HPBRQ2L6CNDUVQLC6E7AOMSTFUOR3HF4DP34LT5Q, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.017835025 (125h48m21.370620154s) [noise=true , address=77.90.31.18:4805 , piece_id=CMDRLWVIYYWWNPZGXPFPPG33CNEVUSUEIWS3ICC475AZDSLUFVCA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:56:46.029952315 (125h48m21.382737453s) [noise=true , address=77.90.31.13:4626 , piece_id=SIRJCFJM5KD36PGP5JW7JRMY7IELU4X7II2MCKAYVY6I5C5YALMA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: <nil>
Sep 3 19:57:15.718514999 (125h48m51.071300107s) [noise=true , address=104.36.82.4:4802 , piece_id=BAOAKJURD3BKKKGJ2L36SNLGZONCWUBQUNS6VKMKALSZIA63APAQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 5.161.89.252:51288->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718576995 (125h48m51.071362094s) [noise=true , address=104.36.82.7:4911 , piece_id=C6R5ZG2G4RROAGRN7GCDKSL23WUMUQFNZAPD7HYVCH2NJHEENHUQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718593346 (125h48m51.071378444s) [noise=true , address=104.36.82.8:4970 , piece_id=VYGUYXA6U22QAK456XS52QEQYLLSO7W3XFFDZYHQ4J6QE4VGOJLA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718626649 (125h48m51.071411747s) [noise=true , address=104.36.82.4:4804 , piece_id=HXMKIDE6KKY4BEWVV23A2JC43LE46EMHTSSUJUKZFEHDFKEGIZWA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718664461 (125h48m51.071449569s) [noise=true , address=104.36.82.6:4874 , piece_id=J2GG4NYMX6BAYOI6F5M54W6CD35I25R3ZU57QHYFSC4NTWW7DNBQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718686712 (125h48m51.071471811s) [noise=true , address=104.36.82.10:4078 , piece_id=FQQLJRJBRCYLEAOSLTBJ7DMEEOS4Y3XTY3OCFQH32DUF6AZFUSSQ, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 5.161.89.252:16156->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718703754 (125h48m51.071488852s) [noise=true , address=104.36.82.10:4077 , piece_id=BXGC2JCKD264PLJAYLBDBMZ5OVILTUMJN3NOCLGAC7HVUXBUXP3A, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718715015 (125h48m51.071500114s) [noise=true , address=104.36.82.9:4006 , piece_id=XKHRBNUTL3B3IZESF7AWOA3XXGZJ7PXAOLHOESYASJROUMHYNZOA, seg_pos={0 0}, upload=110928]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 3 19:57:15.718728822 (125h48m51.07151392s) [seg_pos={0 0}, upload=110928]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Sep 3 19:57:15.731655302 (125h48m51.08444041s) [upload=110928]: Done uploading segments...
Sep 3 19:57:15.731676632 (125h48m51.08446173s) [upload=110928]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 471.031653ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..................X..X.XXX...XX.XX.XX..X...X.X.X.X.......XX.] duration:
624.283908ms
FAIL (2025-09-03 20:21:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 624.304937ms
edge
gateway
us-select-1
|