Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X................X.............X...........] duration:
616.191467ms
FAIL (2025-10-31 21:23:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 616.209842ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X.....................X...] duration:
3.002887995s
FAIL (2025-10-31 21:23:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 3.002907342s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............X...........................................X..] duration:
3.515217303s
FAIL (2025-10-31 21:23:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.515235146s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [...........X.................................X...........X..] duration:
895.894886ms
FAIL (2025-10-31 21:22:59 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 895.919462ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................X............X..........X.] duration:
4.06561297s
FAIL (2025-10-31 21:22:57 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ng segments...
Oct 31 21:21:53.562634555 (106h8m20.26357947s) [upload=93400]: Got next segment. Inline: false
Oct 31 21:21:53.562675572 (106h8m20.263620487s) [upload=93400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:21:53.562681393 (106h8m20.263626308s) [upload=93400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:21:53.585427037 (106h8m20.286371922s) [seg_pos={0 0}, upload=93400]: Begin upload segment...
Oct 31 21:21:53.585488752 (106h8m20.286433638s) [seg_pos={0 0}, upload=93400]: Done begin upload segment.
Oct 31 21:21:53.585512557 (106h8m20.286457452s) [noise=true , address=77.90.31.21:4888 , piece_id=SFCDCLKFMPAYMEHYTGELRHQEJO7TTOPWFGV5L5V4CHNIBOK2YDUA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.585646608 (106h8m20.286591524s) [noise=true , address=77.90.31.16:4746 , piece_id=GUOCWRXD6M5SHQD7IHJRYY4KSYA4W22IC32RGBSRQVJ4Q2W6KV2A, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.586241603 (106h8m20.287186519s) [noise=true , address=45.91.116.12:4252 , piece_id=J4232X5GUD26A4CRFQ3ISC5JEI5UWCASNC7DXN2JVLVIKRP5JIGA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.586833053 (106h8m20.287777968s) [noise=true , address=45.91.116.5:4304 , piece_id=TIF5AWFC6RNRDNTU6SX3TVP6LXJVDJFE4XNCMBAY4JKJLKFMZ6UA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.587291853 (106h8m20.288236778s) [noise=true , address=45.91.116.13:4158 , piece_id=4DCSFW6RPBZDCUAG664KUEF2FOYUD6YBQP74HBGEKO4CBMI6SWLQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.587798323 (106h8m20.288743208s) [noise=true , address=77.90.31.14:4678 , piece_id=NYKOM2GEWWXE4ZCSDQ44YJ6EV3265EE3AMVZLHI2CWFFKPQDEL4Q, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.588370947 (106h8m20.289315842s) [noise=true , address=45.91.116.7:4016 , piece_id=A2AR45B6A65IELR3E22OH2RFUGRRGQWATN3V5AVJSHDEZLW4A4KQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.589171457 (106h8m20.290116373s) [noise=true , address=77.90.31.12:4620 , piece_id=37LL22S666VYKE33N7TVGZQTT47YLAQTHQJ77ZTHLNSA4YNZRTFA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.589510483 (106h8m20.290455369s) [noise=true , address=145.79.227.15:4888 , piece_id=4GKV34FND6K3RQTYXIFGBZOUAE6UW4H47G7G6HTGI3HDSY6BUBBA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.590552788 (106h8m20.291497674s) [noise=true , address=77.90.31.18:4828 , piece_id=5BX4QCRWNOZBZSLMPLTVS3MHI5EPDRCCUHTCOQ3SG6L3YB2EMFUQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.590928573 (106h8m20.291873488s) [noise=true , address=45.91.116.10:4107 , piece_id=MUOH7EE4VOS3SAQ2GWWNTDCHFRSQ6ZG4E5DBXYSN5JDA7TTMZ66A, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.591703135 (106h8m20.292648019s) [noise=true , address=145.79.227.13:4828 , piece_id=TJ3PEJ4AO7T5R43KXEL2TRSP6INHWXU5JKLXJYXHG4QMMRDSZEDA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.591878263 (106h8m20.292823188s) [noise=true , address=45.91.116.6:4977 , piece_id=4UT3BU6LLRKYPSTJDPT237YYM2EQR6LJYV5CBSVXUQYGKSTI2BTA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.592233339 (106h8m20.293178224s) [noise=true , address=145.79.227.16:4931 , piece_id=6Y2BPZS5ACKC3TETCJGOMT7XFQBC7I4RXALBQSDPKBMIS5N5VDLA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.592807466 (106h8m20.293752381s) [noise=true , address=145.79.227.46:4622 , piece_id=ZM5FUON7Q6KZIQMLVWLIBVXAR7J5TFNZNX3GOTFOSMVYFCQDZYQQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.592944223 (106h8m20.293889118s) [noise=true , address=77.90.31.10:4548 , piece_id=L6V2O6O5VZDSVXG2GT6EIKTDDNJTEGLRPYPBMWMWDPZVOUKJ4AKQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.593941883 (106h8m20.294886768s) [noise=true , address=145.79.227.39:4033 , piece_id=K73TAIKN4Z53YSA4CJG4WUHHUPHOVEDTQ2GT7EEHMWPDBYV7DOSQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.594035409 (106h8m20.294980324s) [noise=true , address=145.79.227.51:4413 , piece_id=72N6CVWYSPAHG4GYTMSLRCTX5MCDC7CTSNLVODGP2ZI4MH3SKDUA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.685919559 (106h8m20.386864445s) [upload=93400]: Next returned nil segment
Oct 31 21:21:53.685948013 (106h8m20.386892898s) [upload=93400]: Waiting for error group managing segments...
Oct 31 21:21:53.698433906 (106h8m20.399378831s) [noise=true , address=145.79.227.16:4931 , piece_id=6Y2BPZS5ACKC3TETCJGOMT7XFQBC7I4RXALBQSDPKBMIS5N5VDLA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.706888079 (106h8m20.407832964s) [noise=true , address=145.79.227.39:4033 , piece_id=K73TAIKN4Z53YSA4CJG4WUHHUPHOVEDTQ2GT7EEHMWPDBYV7DOSQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.709310411 (106h8m20.410255337s) [noise=true , address=145.79.227.51:4413 , piece_id=72N6CVWYSPAHG4GYTMSLRCTX5MCDC7CTSNLVODGP2ZI4MH3SKDUA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.720272368 (106h8m20.421217253s) [noise=true , address=145.79.227.13:4828 , piece_id=TJ3PEJ4AO7T5R43KXEL2TRSP6INHWXU5JKLXJYXHG4QMMRDSZEDA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.725700306 (106h8m20.426645231s) [noise=true , address=145.79.227.46:4622 , piece_id=ZM5FUON7Q6KZIQMLVWLIBVXAR7J5TFNZNX3GOTFOSMVYFCQDZYQQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.728322684 (106h8m20.429267569s) [noise=true , address=145.79.227.15:4888 , piece_id=4GKV34FND6K3RQTYXIFGBZOUAE6UW4H47G7G6HTGI3HDSY6BUBBA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:31.360110266 (106h8m58.061055161s) [noise=true , address=77.90.31.10:4548 , piece_id=L6V2O6O5VZDSVXG2GT6EIKTDDNJTEGLRPYPBMWMWDPZVOUKJ4AKQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:34.072462431 (106h9m0.773407326s) [noise=true , address=45.91.116.6:4977 , piece_id=4UT3BU6LLRKYPSTJDPT237YYM2EQR6LJYV5CBSVXUQYGKSTI2BTA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:37.283451084 (106h9m3.984395979s) [noise=true , address=77.90.31.12:4620 , piece_id=37LL22S666VYKE33N7TVGZQTT47YLAQTHQJ77ZTHLNSA4YNZRTFA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:40.683478084 (106h9m7.384422999s) [noise=true , address=45.91.116.10:4107 , piece_id=MUOH7EE4VOS3SAQ2GWWNTDCHFRSQ6ZG4E5DBXYSN5JDA7TTMZ66A, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:44.017006527 (106h9m10.717951422s) [noise=true , address=77.90.31.16:4746 , piece_id=GUOCWRXD6M5SHQD7IHJRYY4KSYA4W22IC32RGBSRQVJ4Q2W6KV2A, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:45.961297890 (106h9m12.662242815s) [noise=true , address=45.91.116.7:4016 , piece_id=A2AR45B6A65IELR3E22OH2RFUGRRGQWATN3V5AVJSHDEZLW4A4KQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:46.915892770 (106h9m13.616837665s) [noise=true , address=77.90.31.14:4678 , piece_id=NYKOM2GEWWXE4ZCSDQ44YJ6EV3265EE3AMVZLHI2CWFFKPQDEL4Q, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:53.549094354 (106h9m20.250039249s) [noise=true , address=77.90.31.18:4828 , piece_id=5BX4QCRWNOZBZSLMPLTVS3MHI5EPDRCCUHTCOQ3SG6L3YB2EMFUQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:22:53.549174364 (106h9m20.25011925s) [noise=true , address=77.90.31.21:4888 , piece_id=SFCDCLKFMPAYMEHYTGELRHQEJO7TTOPWFGV5L5V4CHNIBOK2YDUA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:12BiqwcDuvobZabx3jxDhXf27Ei8oLFS6wLp3qbpZCtJ4ZhQEZA, address:77.90.31.21:4888): protocol: write tcp 5.161.89.252:33992->77.90.31.21:4888: use of closed network connection; write tcp 5.161.89.252:33992->77.90.31.21:4888: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:33992->77.90.31.21:4888: use of closed network connection
Oct 31 21:22:53.549300751 (106h9m20.250245647s) [noise=true , address=45.91.116.12:4252 , piece_id=J4232X5GUD26A4CRFQ3ISC5JEI5UWCASNC7DXN2JVLVIKRP5JIGA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: read tcp 5.161.89.252:26998->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:53.549318845 (106h9m20.250263761s) [noise=true , address=45.91.116.13:4158 , piece_id=4DCSFW6RPBZDCUAG664KUEF2FOYUD6YBQP74HBGEKO4CBMI6SWLQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:1f9tm8dBDjspqLBwSmXeyBgxm8H1aonRUjbWcvLfxcb52JaX6a, address:45.91.116.13:4158): protocol: read tcp 5.161.89.252:22182->45.91.116.13:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:53.549531584 (106h9m20.250476509s) [noise=true , address=45.91.116.5:4304 , piece_id=TIF5AWFC6RNRDNTU6SX3TVP6LXJVDJFE4XNCMBAY4JKJLKFMZ6UA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:12dFsmPBv2oP7W7uKnJFpKMcJDSQ3BEGLaPZ23eX3Cqze8UgDSs, address:45.91.116.5:4304): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:22:53.549563313 (106h9m20.250508229s) [seg_pos={0 0}, upload=93400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:22:53.570132083 (106h9m20.271076978s) [upload=93400]: Done uploading segments...
Oct 31 21:22:53.570144777 (106h9m20.271089672s) [upload=93400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 4.065635903s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X................X.............X..........] duration:
10.368850209s
FAIL (2025-10-31 21:22:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.368866019s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............X...........................................X.] duration:
15.840858086s
FAIL (2025-10-31 21:22:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.840980906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................X............X..........X] duration:
1m0.022917795s
FAIL (2025-10-31 21:22:53 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ng segments...
Oct 31 21:21:53.562634555 (106h8m20.26357947s) [upload=93400]: Got next segment. Inline: false
Oct 31 21:21:53.562675572 (106h8m20.263620487s) [upload=93400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:21:53.562681393 (106h8m20.263626308s) [upload=93400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:21:53.585427037 (106h8m20.286371922s) [seg_pos={0 0}, upload=93400]: Begin upload segment...
Oct 31 21:21:53.585488752 (106h8m20.286433638s) [seg_pos={0 0}, upload=93400]: Done begin upload segment.
Oct 31 21:21:53.585512557 (106h8m20.286457452s) [noise=true , address=77.90.31.21:4888 , piece_id=SFCDCLKFMPAYMEHYTGELRHQEJO7TTOPWFGV5L5V4CHNIBOK2YDUA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.585646608 (106h8m20.286591524s) [noise=true , address=77.90.31.16:4746 , piece_id=GUOCWRXD6M5SHQD7IHJRYY4KSYA4W22IC32RGBSRQVJ4Q2W6KV2A, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.586241603 (106h8m20.287186519s) [noise=true , address=45.91.116.12:4252 , piece_id=J4232X5GUD26A4CRFQ3ISC5JEI5UWCASNC7DXN2JVLVIKRP5JIGA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.586833053 (106h8m20.287777968s) [noise=true , address=45.91.116.5:4304 , piece_id=TIF5AWFC6RNRDNTU6SX3TVP6LXJVDJFE4XNCMBAY4JKJLKFMZ6UA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.587291853 (106h8m20.288236778s) [noise=true , address=45.91.116.13:4158 , piece_id=4DCSFW6RPBZDCUAG664KUEF2FOYUD6YBQP74HBGEKO4CBMI6SWLQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.587798323 (106h8m20.288743208s) [noise=true , address=77.90.31.14:4678 , piece_id=NYKOM2GEWWXE4ZCSDQ44YJ6EV3265EE3AMVZLHI2CWFFKPQDEL4Q, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.588370947 (106h8m20.289315842s) [noise=true , address=45.91.116.7:4016 , piece_id=A2AR45B6A65IELR3E22OH2RFUGRRGQWATN3V5AVJSHDEZLW4A4KQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.589171457 (106h8m20.290116373s) [noise=true , address=77.90.31.12:4620 , piece_id=37LL22S666VYKE33N7TVGZQTT47YLAQTHQJ77ZTHLNSA4YNZRTFA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.589510483 (106h8m20.290455369s) [noise=true , address=145.79.227.15:4888 , piece_id=4GKV34FND6K3RQTYXIFGBZOUAE6UW4H47G7G6HTGI3HDSY6BUBBA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.590552788 (106h8m20.291497674s) [noise=true , address=77.90.31.18:4828 , piece_id=5BX4QCRWNOZBZSLMPLTVS3MHI5EPDRCCUHTCOQ3SG6L3YB2EMFUQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.590928573 (106h8m20.291873488s) [noise=true , address=45.91.116.10:4107 , piece_id=MUOH7EE4VOS3SAQ2GWWNTDCHFRSQ6ZG4E5DBXYSN5JDA7TTMZ66A, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.591703135 (106h8m20.292648019s) [noise=true , address=145.79.227.13:4828 , piece_id=TJ3PEJ4AO7T5R43KXEL2TRSP6INHWXU5JKLXJYXHG4QMMRDSZEDA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.591878263 (106h8m20.292823188s) [noise=true , address=45.91.116.6:4977 , piece_id=4UT3BU6LLRKYPSTJDPT237YYM2EQR6LJYV5CBSVXUQYGKSTI2BTA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.592233339 (106h8m20.293178224s) [noise=true , address=145.79.227.16:4931 , piece_id=6Y2BPZS5ACKC3TETCJGOMT7XFQBC7I4RXALBQSDPKBMIS5N5VDLA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.592807466 (106h8m20.293752381s) [noise=true , address=145.79.227.46:4622 , piece_id=ZM5FUON7Q6KZIQMLVWLIBVXAR7J5TFNZNX3GOTFOSMVYFCQDZYQQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.592944223 (106h8m20.293889118s) [noise=true , address=77.90.31.10:4548 , piece_id=L6V2O6O5VZDSVXG2GT6EIKTDDNJTEGLRPYPBMWMWDPZVOUKJ4AKQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.593941883 (106h8m20.294886768s) [noise=true , address=145.79.227.39:4033 , piece_id=K73TAIKN4Z53YSA4CJG4WUHHUPHOVEDTQ2GT7EEHMWPDBYV7DOSQ, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.594035409 (106h8m20.294980324s) [noise=true , address=145.79.227.51:4413 , piece_id=72N6CVWYSPAHG4GYTMSLRCTX5MCDC7CTSNLVODGP2ZI4MH3SKDUA, seg_pos={0 0}, upload=93400]: Uploading piece...
Oct 31 21:21:53.685919559 (106h8m20.386864445s) [upload=93400]: Next returned nil segment
Oct 31 21:21:53.685948013 (106h8m20.386892898s) [upload=93400]: Waiting for error group managing segments...
Oct 31 21:21:53.698433906 (106h8m20.399378831s) [noise=true , address=145.79.227.16:4931 , piece_id=6Y2BPZS5ACKC3TETCJGOMT7XFQBC7I4RXALBQSDPKBMIS5N5VDLA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.706888079 (106h8m20.407832964s) [noise=true , address=145.79.227.39:4033 , piece_id=K73TAIKN4Z53YSA4CJG4WUHHUPHOVEDTQ2GT7EEHMWPDBYV7DOSQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.709310411 (106h8m20.410255337s) [noise=true , address=145.79.227.51:4413 , piece_id=72N6CVWYSPAHG4GYTMSLRCTX5MCDC7CTSNLVODGP2ZI4MH3SKDUA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.720272368 (106h8m20.421217253s) [noise=true , address=145.79.227.13:4828 , piece_id=TJ3PEJ4AO7T5R43KXEL2TRSP6INHWXU5JKLXJYXHG4QMMRDSZEDA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.725700306 (106h8m20.426645231s) [noise=true , address=145.79.227.46:4622 , piece_id=ZM5FUON7Q6KZIQMLVWLIBVXAR7J5TFNZNX3GOTFOSMVYFCQDZYQQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:21:53.728322684 (106h8m20.429267569s) [noise=true , address=145.79.227.15:4888 , piece_id=4GKV34FND6K3RQTYXIFGBZOUAE6UW4H47G7G6HTGI3HDSY6BUBBA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:31.360110266 (106h8m58.061055161s) [noise=true , address=77.90.31.10:4548 , piece_id=L6V2O6O5VZDSVXG2GT6EIKTDDNJTEGLRPYPBMWMWDPZVOUKJ4AKQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:34.072462431 (106h9m0.773407326s) [noise=true , address=45.91.116.6:4977 , piece_id=4UT3BU6LLRKYPSTJDPT237YYM2EQR6LJYV5CBSVXUQYGKSTI2BTA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:37.283451084 (106h9m3.984395979s) [noise=true , address=77.90.31.12:4620 , piece_id=37LL22S666VYKE33N7TVGZQTT47YLAQTHQJ77ZTHLNSA4YNZRTFA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:40.683478084 (106h9m7.384422999s) [noise=true , address=45.91.116.10:4107 , piece_id=MUOH7EE4VOS3SAQ2GWWNTDCHFRSQ6ZG4E5DBXYSN5JDA7TTMZ66A, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:44.017006527 (106h9m10.717951422s) [noise=true , address=77.90.31.16:4746 , piece_id=GUOCWRXD6M5SHQD7IHJRYY4KSYA4W22IC32RGBSRQVJ4Q2W6KV2A, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:45.961297890 (106h9m12.662242815s) [noise=true , address=45.91.116.7:4016 , piece_id=A2AR45B6A65IELR3E22OH2RFUGRRGQWATN3V5AVJSHDEZLW4A4KQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:46.915892770 (106h9m13.616837665s) [noise=true , address=77.90.31.14:4678 , piece_id=NYKOM2GEWWXE4ZCSDQ44YJ6EV3265EE3AMVZLHI2CWFFKPQDEL4Q, seg_pos={0 0}, upload=93400]: Done uploading piece. err: <nil>
Oct 31 21:22:53.549094354 (106h9m20.250039249s) [noise=true , address=77.90.31.18:4828 , piece_id=5BX4QCRWNOZBZSLMPLTVS3MHI5EPDRCCUHTCOQ3SG6L3YB2EMFUQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:22:53.549174364 (106h9m20.25011925s) [noise=true , address=77.90.31.21:4888 , piece_id=SFCDCLKFMPAYMEHYTGELRHQEJO7TTOPWFGV5L5V4CHNIBOK2YDUA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:12BiqwcDuvobZabx3jxDhXf27Ei8oLFS6wLp3qbpZCtJ4ZhQEZA, address:77.90.31.21:4888): protocol: write tcp 5.161.89.252:33992->77.90.31.21:4888: use of closed network connection; write tcp 5.161.89.252:33992->77.90.31.21:4888: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:33992->77.90.31.21:4888: use of closed network connection
Oct 31 21:22:53.549300751 (106h9m20.250245647s) [noise=true , address=45.91.116.12:4252 , piece_id=J4232X5GUD26A4CRFQ3ISC5JEI5UWCASNC7DXN2JVLVIKRP5JIGA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: read tcp 5.161.89.252:26998->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:53.549318845 (106h9m20.250263761s) [noise=true , address=45.91.116.13:4158 , piece_id=4DCSFW6RPBZDCUAG664KUEF2FOYUD6YBQP74HBGEKO4CBMI6SWLQ, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:1f9tm8dBDjspqLBwSmXeyBgxm8H1aonRUjbWcvLfxcb52JaX6a, address:45.91.116.13:4158): protocol: read tcp 5.161.89.252:22182->45.91.116.13:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:53.549531584 (106h9m20.250476509s) [noise=true , address=45.91.116.5:4304 , piece_id=TIF5AWFC6RNRDNTU6SX3TVP6LXJVDJFE4XNCMBAY4JKJLKFMZ6UA, seg_pos={0 0}, upload=93400]: Done uploading piece. err: ecclient: upload failed (node:12dFsmPBv2oP7W7uKnJFpKMcJDSQ3BEGLaPZ23eX3Cqze8UgDSs, address:45.91.116.5:4304): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:22:53.549563313 (106h9m20.250508229s) [seg_pos={0 0}, upload=93400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 31 21:22:53.570132083 (106h9m20.271076978s) [upload=93400]: Done uploading segments...
Oct 31 21:22:53.570144777 (106h9m20.271089672s) [upload=93400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.022932672s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X.....................X..] duration:
1.42617782s
FAIL (2025-10-31 21:22:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 1.426202877s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................X................X.............X.........] duration:
20.135116056s
FAIL (2025-10-31 21:22:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.135189684s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............X...........................................X] duration:
1m0.001416819s
FAIL (2025-10-31 21:22:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001432628s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X.....................X.] duration:
471.889935ms
FAIL (2025-10-31 21:22:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 471.906305ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X.....................X] duration:
30.022635769s
FAIL (2025-10-31 21:22:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:22:00.944281574 (106h8m27.645226459s) [upload=93401]: Starting upload
Oct 31 21:22:00.944356304 (106h8m27.645301199s) [upload=93401]: Done starting upload
Oct 31 21:22:00.944468063 (106h8m27.645412948s) [upload=93401]: Uploading segments...
Oct 31 21:22:00.944630548 (106h8m27.645575433s) [upload=93401]: Got next segment. Inline: false
Oct 31 21:22:00.944665213 (106h8m27.645610108s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:00.944670022 (106h8m27.645614907s) [upload=93401]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:00.963897347 (106h8m27.664842243s) [seg_pos={0 0}, upload=93401]: Begin upload segment...
Oct 31 21:22:00.963998978 (106h8m27.664943873s) [seg_pos={0 0}, upload=93401]: Done begin upload segment.
Oct 31 21:22:00.964028473 (106h8m27.664973368s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964037450 (106h8m27.664982345s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964444233 (106h8m27.665389128s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.964892844 (106h8m27.665837739s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965325144 (106h8m27.666270039s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.965625197 (106h8m27.666570082s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966019857 (106h8m27.666964752s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966370855 (106h8m27.66731574s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.966687539 (106h8m27.667632424s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.967022156 (106h8m27.667967042s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.968385533 (106h8m27.669330428s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.969631550 (106h8m27.670576435s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.971122787 (106h8m27.672067672s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.972500710 (106h8m27.673445595s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.973763789 (106h8m27.674708684s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:00.974994406 (106h8m27.675939291s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Uploading piece...
Oct 31 21:22:01.045346730 (106h8m27.746291626s) [upload=93401]: Next returned nil segment
Oct 31 21:22:01.045360827 (106h8m27.746305712s) [upload=93401]: Waiting for error group managing segments...
Oct 31 21:22:01.055543281 (106h8m27.756488176s) [noise=true , address=145.79.227.10:4403 , piece_id=LWQNMRJGDAPJX32YTJROJZHNXG43WGD654XYL7EWBERGZDTD24EA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055753104 (106h8m27.75669798s) [noise=true , address=145.79.227.4:4185 , piece_id=SKOREBXEU6FYVD5QQRMUX6OXKSOOZ23U2AKPOI6W76WEJKXZFH6Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.055999196 (106h8m27.756944081s) [noise=true , address=145.79.227.9:4367 , piece_id=H4Y3RSE5LDYMDMIJSVG2OZ2ZFQIDINZ5LGTJ6G6GRJMQOVISMVZQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056605764 (106h8m27.757550649s) [noise=true , address=145.79.227.31:4798 , piece_id=WOB4HFBOQXCSZNWCX6KCOLWS5VD7OWTVCQV35OJBKVPO6HIMJCFQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.056939789 (106h8m27.757884675s) [noise=true , address=145.79.227.31:4797 , piece_id=J3E54GKYZ2NAQ2A27VCTRZ6K3RUSTINK3TB4JMIXU4GSWDE6R7VA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057268406 (106h8m27.758213291s) [noise=true , address=145.79.227.11:4730 , piece_id=KNGY3NFHFXX76TGLHY7NYOKJMFB2HHH4BMUAQAKEVP5PWW4Q5SMQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057510931 (106h8m27.758455815s) [noise=true , address=145.79.227.32:4835 , piece_id=EWXGE5HU7KV73PTPTGGTS6J7FFGI5VKVGLD4CPSC52DV5MBJQS2Q, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.057806124 (106h8m27.758751009s) [noise=true , address=145.79.227.30:4762 , piece_id=XOOSZCH5AJ45I376ARJ6QFCOCKZZOUHO35IM4JTXFMMJHCY2MDLQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.058699179 (106h8m27.759644064s) [noise=true , address=145.79.227.27:4691 , piece_id=BEVJVVTJN4N5VVSOFJPRIZRQK6XWA36KIGW73OYXYADKMRJ2UBPQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059036802 (106h8m27.759981688s) [noise=true , address=145.79.227.21:4473 , piece_id=CI63WAVFZ5NUWE4IQPX27OVGOUBBVT5K25QB3JSIB3R6KDO2OZWA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:01.059484752 (106h8m27.760429636s) [noise=true , address=145.79.227.13:4803 , piece_id=VNVRU2HCQ6EMSEE3JLIEQAJATEAGK6FXHQCHGL3XITBUQB66K2YA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: <nil>
Oct 31 21:22:30.929275627 (106h8m57.630220522s) [noise=true , address=77.90.31.13:4626 , piece_id=XHF3BKN6IX4T6DS5O7R2KQL7N3ZGXS5PM57MDJ2TRSZVJVM7Y2YQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929342072 (106h8m57.630286956s) [noise=true , address=45.91.116.4:4939 , piece_id=P6I6RHCEKQAHJZJDRQTQZVUB2RKGLOS6MB4SKYAWM6DMMLI2GGKA, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 5.161.89.252:31798->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929405521 (106h8m57.630350416s) [noise=true , address=77.90.31.18:4805 , piece_id=WYUSUGKCOEKXTYOS6RPZWDGXXW6TUP6B443KERADCBBJIO2M2E7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929460374 (106h8m57.630405269s) [noise=true , address=77.90.31.17:4770 , piece_id=A3VGNQ25LGB6AQEDFXSIB7HWLV546CUPKYRYJFAVWBSRIVGG2FEQ, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 5.161.89.252:53362->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:30.929546896 (106h8m57.630491791s) [noise=true , address=77.90.31.19:4841 , piece_id=DHNCAWEX7CTYBTIG4IDB6R4MXUXLRMKN4TFNPBOUDZV5FMAKRM7A, seg_pos={0 0}, upload=93401]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: read tcp 5.161.89.252:26862->77.90.31.19:4841: use of closed network connection; context deadline exceeded
Oct 31 21:22:30.929573285 (106h8m57.63051818s) [seg_pos={0 0}, upload=93401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:22:30.950873398 (106h8m57.651818293s) [upload=93401]: Done uploading segments...
Oct 31 21:22:30.950890520 (106h8m57.651835415s) [upload=93401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.022655457s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [............X.................................X...........X.] duration:
2.991132505s
FAIL (2025-10-31 21:22:04 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.991144847s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=3) [.............X.................................X...........X] duration:
5.001595001s
FAIL (2025-10-31 21:21:46 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001636999s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................X................X.............X........] duration:
342.254181ms
FAIL (2025-10-31 21:21:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 342.271063ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
510.367158ms
FAIL (2025-10-31 21:21:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 510.392034ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [..............X.................................X...........] duration:
515.663138ms
FAIL (2025-10-31 21:21:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 515.679769ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X............X..........] duration:
460.446676ms
FAIL (2025-10-31 21:21:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 460.495778ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
625.861359ms
FAIL (2025-10-31 21:20:51 UTC)
orbiter warning: context deadline exceeded
Duration: 625.878121ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
698.520181ms
FAIL (2025-10-31 21:20:32 UTC)
orbiter warning: context deadline exceeded
Duration: 698.537442ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
395.590369ms
FAIL (2025-10-31 21:20:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 395.62769ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................X................X.............X.......] duration:
375.494423ms
FAIL (2025-10-31 21:20:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 375.511676ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
661.058832ms
FAIL (2025-10-31 21:20:06 UTC)
orbiter warning: context deadline exceeded
Duration: 661.075645ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X............X.........] duration:
467.773475ms
FAIL (2025-10-31 21:19:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 467.796468ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [...............X.................................X..........] duration:
527.761694ms
FAIL (2025-10-31 21:19:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 527.781572ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X............X........] duration:
436.34936ms
FAIL (2025-10-31 21:19:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 436.368165ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X................X.............X......] duration:
377.995936ms
FAIL (2025-10-31 21:19:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 378.015882ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
440.993137ms
FAIL (2025-10-31 21:19:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 441.010951ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [................X.................................X.........] duration:
545.384171ms
FAIL (2025-10-31 21:18:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 545.40027ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
675.522135ms
FAIL (2025-10-31 21:18:59 UTC)
orbiter warning: context deadline exceeded
Duration: 675.55759ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
328.578779ms
FAIL (2025-10-31 21:18:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 328.609937ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X............X.......] duration:
583.414355ms
FAIL (2025-10-31 21:18:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 583.434383ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X................X.............X.....] duration:
344.794436ms
FAIL (2025-10-31 21:18:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 344.813401ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [.................X.................................X........] duration:
549.269055ms
FAIL (2025-10-31 21:18:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 549.28214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X............X......] duration:
552.370882ms
FAIL (2025-10-31 21:18:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 552.395597ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
608.061462ms
FAIL (2025-10-31 21:17:59 UTC)
orbiter warning: context deadline exceeded
Duration: 608.079706ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
429.537677ms
FAIL (2025-10-31 21:17:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 429.562895ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................X................X.............X....] duration:
362.587292ms
FAIL (2025-10-31 21:17:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 362.603503ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [..................X.................................X.......] duration:
510.793608ms
FAIL (2025-10-31 21:17:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.809838ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [...................X.................................X......] duration:
542.729631ms
FAIL (2025-10-31 21:17:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 542.806746ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X............X.....] duration:
417.344752ms
FAIL (2025-10-31 21:17:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 417.369367ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
363.683508ms
FAIL (2025-10-31 21:17:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
[upload=93136]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:22.790402507 (105h49m49.491347402s) [upload=93136]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:22.812634163 (105h49m49.513579048s) [seg_pos={0 0}, upload=93136]: Begin upload segment...
Oct 31 21:03:22.812679268 (105h49m49.513624152s) [seg_pos={0 0}, upload=93136]: Done begin upload segment.
Oct 31 21:03:22.812705407 (105h49m49.513650292s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.812748467 (105h49m49.513693352s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813185797 (105h49m49.514130702s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813307676 (105h49m49.514252571s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813481142 (105h49m49.514426016s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813689412 (105h49m49.514634297s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813752691 (105h49m49.514697576s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.813997220 (105h49m49.514942094s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814086856 (105h49m49.515031741s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814259240 (105h49m49.515204125s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814486787 (105h49m49.515431672s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814524668 (105h49m49.515469543s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814785337 (105h49m49.515730221s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.814848044 (105h49m49.515792929s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815249898 (105h49m49.516194783s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.815718176 (105h49m49.516663081s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Uploading piece...
Oct 31 21:03:22.892458609 (105h49m49.593403495s) [upload=93136]: Next returned nil segment
Oct 31 21:03:22.892473317 (105h49m49.593418212s) [upload=93136]: Waiting for error group managing segments...
Oct 31 21:03:22.902106631 (105h49m49.603051515s) [noise=true , address=145.79.227.15:4874 , piece_id=HGHY7VQWIX3DRNBUYUGKEWUYYPKZEOBWPHZJ27KMDTJ3JCQGHZOA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902371287 (105h49m49.603316182s) [noise=true , address=145.79.227.4:4185 , piece_id=HUBJ7QLCBQCW4VB6QAQFOE5IFMIFQFZJUMNCFQ5SV2DHQRCXM4YA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.902429837 (105h49m49.603374721s) [noise=true , address=145.79.227.21:4473 , piece_id=MPGESJN55C54UEKPWPODRTDKRFKRTMDIXAXH5GNMRVTMRFGK7A2Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903642190 (105h49m49.604587074s) [noise=true , address=145.79.227.10:4401 , piece_id=BLVTGVW4UPEIOFQLPJ54CBNE7K542KHUAGE43QL2AHROYOIUTBVQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.903872101 (105h49m49.604816985s) [noise=true , address=145.79.227.14:4837 , piece_id=MBHBKN6IOGCA37QLGATBXORLSI363WJMPZOCDIB5NIDJYZUFV5FQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904246663 (105h49m49.605191548s) [noise=true , address=145.79.227.30:4763 , piece_id=7YXBVB3KKO77KLWY3RIC77P726M4NWY4QM5FQQ7QLNBLTDPV6Y4A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904300284 (105h49m49.605245169s) [noise=true , address=145.79.227.31:4798 , piece_id=HCUKC5FN46V2IP43NXR5S2OGTYWADIXISKU2PI46PYLDPAZDYM6Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.904611247 (105h49m49.605556142s) [noise=true , address=145.79.227.27:4690 , piece_id=4PVRYCMUZ6QNNWY6WHRVYYWJUMZ4HS6DXECTO3ALVNJSNA4T3AYQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:22.905597115 (105h49m49.606542s) [noise=true , address=145.79.227.27:4689 , piece_id=BZKXCAMFUHFQFEIVAPKBKURBWNGWVENHXS72T3DLXSRGJ5TLQJXA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: <nil>
Oct 31 21:03:52.778803424 (105h50m19.479748309s) [noise=true , address=45.91.116.4:4940 , piece_id=KKBVAPRBHUUZNCNJICQVQ3G7QBR5N567YNZZBWGHWHAROK2BWV5Q, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:56452->45.91.116.4:4940: use of closed network connection
Oct 31 21:03:52.778865040 (105h50m19.479809935s) [noise=true , address=45.91.116.4:4938 , piece_id=4XZYGRVTLIOBZJCHNG6R7GX36JDMQN6OM4BO5HP2JVILPKNLRUGA, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:41356->45.91.116.4:4938: use of closed network connection
Oct 31 21:03:52.778945942 (105h50m19.479890827s) [noise=true , address=77.90.31.19:4842 , piece_id=IPEB2DL7WDCAQCGIPJYHTVMQF3AL65XOPSMUBN4CG5XZKFYFIZZQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 5.161.89.252:46822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779019018 (105h50m19.479963913s) [noise=true , address=77.90.31.15:4698 , piece_id=5KBIBQ74S5BHUIMSWOZOV6SBIQW4ZHCFWIDHYG2VHMF457I5UXNQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 5.161.89.252:54344->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779027594 (105h50m19.479972489s) [noise=true , address=45.91.116.8:4045 , piece_id=QAHGSIPMDS5OKWQTKGUWHRAQHDKDIXXEQAEG4YQXMKYC74V2H6AQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 5.161.89.252:20860->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:52.779046510 (105h50m19.479991395s) [noise=true , address=77.90.31.16:4735 , piece_id=7VPQGWC5MDRSY26DFQODS6QCAPHNW3PCLYD3JESL6GBRQRZBQP5A, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:20562->77.90.31.16:4735: use of closed network connection
Oct 31 21:03:52.779155314 (105h50m19.480100209s) [noise=true , address=77.90.31.18:4805 , piece_id=WSCTLK7BZ5Y2GIP4FWBMA6O56V7XAS4EG3X6UYBKGDBSPPY3XUBQ, seg_pos={0 0}, upload=93136]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 5.161.89.252:33814->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 31 21:03:52.779180812 (105h50m19.480125697s) [seg_pos={0 0}, upload=93136]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 31 21:03:52.797905573 (105h50m19.498850458s) [upload=93136]: Done uploading segments...
Oct 31 21:03:52.797925189 (105h50m19.498870074s) [upload=93136]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 363.71216ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................X................X.............X...] duration:
390.79681ms
FAIL (2025-10-31 21:16:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 390.820995ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
592.097307ms
FAIL (2025-10-31 21:16:48 UTC)
orbiter warning: context deadline exceeded
Duration: 592.117095ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [....................X.................................X.....] duration:
512.509113ms
FAIL (2025-10-31 21:16:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 512.531686ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X............X....] duration:
421.346826ms
FAIL (2025-10-31 21:16:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489220300 (105h59m54.190165196s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.489559817 (105h59m54.190504703s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490016915 (105h59m54.19096181s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490431101 (105h59m54.191375996s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.490921210 (105h59m54.191866105s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491320298 (105h59m54.192265184s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491637283 (105h59m54.192582168s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.491993701 (105h59m54.192938596s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492347455 (105h59m54.19329234s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.492684076 (105h59m54.193628971s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.494314954 (105h59m54.195259849s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Uploading piece...
Oct 31 21:13:27.576610724 (105h59m54.27755561s) [upload=93273]: Next returned nil segment
Oct 31 21:13:27.576639588 (105h59m54.277584464s) [upload=93273]: Waiting for error group managing segments...
Oct 31 21:13:27.601581249 (105h59m54.302526154s) [noise=true , address=145.79.227.49:4338 , piece_id=77SWVJ2SG3EQRLAWLWKGABG2VFXCN7YLRTKTLXX4ECVMCEEYQ4TA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.604395536 (105h59m54.305340441s) [noise=true , address=145.79.227.12:4780 , piece_id=SJCPSX4ERHN6353ZMKF7DS5J6IQ54ANUZJCMMDWQXAAQOKB4US4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.613295836 (105h59m54.314240721s) [noise=true , address=145.79.227.20:4470 , piece_id=4QX6LCUEX5RL4JE6QBZTDYSVQGEOJMDRRUCGJJBSEJ2WL7KDUT6Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.619255753 (105h59m54.320200648s) [noise=true , address=145.79.227.4:4211 , piece_id=BGTGHX22JS7DKYEUCCRTCT6QRUSCZ3RHP5KBZ3D2WTHH5E3OIKLQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.621122162 (105h59m54.322067057s) [noise=true , address=145.79.227.17:4980 , piece_id=YFU2APCK2H34O47J6XNMWTLLN5WIZIU4ZIK4UJHLZJPAZP23KQ4Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:27.633764760 (105h59m54.334709674s) [noise=true , address=145.79.227.40:4062 , piece_id=XF7WFJTDP6HZGE6675FEOO4X76ZMDH7LHELER3IOVVRN5FCL3GEA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:13:30.536247637 (105h59m57.237192521s) [noise=true , address=145.79.227.16:4928 , piece_id=IOUDHEZTRV6WVGXPSXWM7H4QZQTTYLWCTJKZ7B4UWARCUPR7LEJA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: <nil>
Oct 31 21:14:27.438813050 (106h0m54.139757945s) [noise=true , address=45.91.116.5:4308 , piece_id=YN7UNDUXXJXH6EYMIJT2TXROXT4Q2XVS4YJ2K6RBPWXSIMS7DVPA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4, address:45.91.116.5:4308): protocol: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:12694->45.91.116.5:4308: use of closed network connection
Oct 31 21:14:27.438927465 (106h0m54.13987235s) [noise=true , address=45.91.116.8:4063 , piece_id=CHUULL4OPAIY2537T3LSHA56SKK3J5BZ3UCPRJQXP7DAQESB24VA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.438957591 (106h0m54.139902476s) [noise=true , address=77.90.31.16:4750 , piece_id=JLCX7CATMWDOI2AGTLAKWG3DEXILT6MZQOY5MDSHTS62ID2C2Z5Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 5.161.89.252:50622->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439074560 (106h0m54.140019455s) [noise=true , address=45.91.116.10:4091 , piece_id=7I45ZOZE5VZB3U4KS4T7P2JPK23SSROZADKZY45PSYNSMESNPUHQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:14:27.439151815 (106h0m54.140096701s) [noise=true , address=77.90.31.10:4552 , piece_id=ZDOPX7KZYCR3TJHEIREPXRKPRD53OC65LFOIVD7PXCMGCG7XXBXA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:58508->77.90.31.10:4552: use of closed network connection
Oct 31 21:14:27.439251072 (106h0m54.140195977s) [noise=true , address=45.91.116.4:4947 , piece_id=A6XZC7CYKA4MPNBYOL63R2IGRVBF3SZCRTBG3WSAJSCOUDFPKO7A, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YTkM1w6HKKyw8z3eWomLMtJuFxsswqjZ5TmGyBeiZJE9QrFoB, address:45.91.116.4:4947): protocol: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:11834->45.91.116.4:4947: use of closed network connection
Oct 31 21:14:27.439274105 (106h0m54.140218989s) [noise=true , address=45.91.116.8:4059 , piece_id=H2P3IEUEAWRIRW7U3BZIQJKXO7FKBSXCYIQAIPRF2ZWDPVL4JAQQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1YzYcwyVJqZywPJt9tYXXHMrjfjeziu6fobhHjt6sRf3utaror, address:45.91.116.8:4059): protocol: rpc: dial tcp 45.91.116.8:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439365316 (106h0m54.140310201s) [noise=true , address=45.91.116.7:4044 , piece_id=3JDOIZZXIUJUBZM4GH7WJ5ZKEETCZVOTY36YPLDRA5RSVJVCWYZQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz, address:45.91.116.7:4044): protocol: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection; piecestore: piecestore close: write tcp 5.161.89.252:15056->45.91.116.7:4044: use of closed network connection
Oct 31 21:14:27.439469882 (106h0m54.140414787s) [noise=true , address=77.90.31.18:4816 , piece_id=3MRPH36RULL4GAR3JKSAKNRWNHXZTUUZNMY77KWU6BFCTKRVAS7Q, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 5.161.89.252:52750->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439479670 (106h0m54.140424555s) [noise=true , address=77.90.31.13:4648 , piece_id=52GP7EWMAMJCBVNRYFNV6IOZN3ZZKSBOKSGMSP3CKOKODGOAOKHA, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: read tcp 5.161.89.252:52818->77.90.31.13:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439557907 (106h0m54.140502802s) [noise=true , address=77.90.31.19:4868 , piece_id=V56GLWAOB2WXODPZVKPFFPQWF4EVV7FM3F5LXLQI3RIFTZ7VKRIQ, seg_pos={0 0}, upload=93273]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: read tcp 5.161.89.252:31250->77.90.31.19:4868: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:27.439574608 (106h0m54.140519503s) [seg_pos={0 0}, upload=93273]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Oct 31 21:14:27.456689141 (106h0m54.157634026s) [upload=93273]: Done uploading segments...
Oct 31 21:14:27.456707465 (106h0m54.15765236s) [upload=93273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 421.372355ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [.....................X.................................X....] duration:
519.847256ms
FAIL (2025-10-31 21:16:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 519.901669ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
839.486749ms
FAIL (2025-10-31 21:16:11 UTC)
orbiter warning: context deadline exceeded
Duration: 839.511706ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=3) [......................X.................................X...] duration:
542.539158ms
FAIL (2025-10-31 21:16:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 542.557653ms
core
uplink
us-select-1
us1-placement-20
|