Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [X.XX..XX....................................................] duration:
500.989323ms
FAIL (2025-11-22 13:10:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.011133ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
397.763261ms
FAIL (2025-11-22 13:10:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 397.791934ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
330.810573ms
FAIL (2025-11-22 13:10:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 330.825712ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
550.9194ms
FAIL (2025-11-22 13:10:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 550.962752ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration:
366.918465ms
FAIL (2025-11-22 13:10:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 366.944945ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [.X.XX..XX...................................................] duration:
453.83843ms
FAIL (2025-11-22 13:10:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 453.863005ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration:
477.700952ms
FAIL (2025-11-22 13:09:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 477.72134ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
603.080203ms
FAIL (2025-11-22 13:09:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 603.094579ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
362.340024ms
FAIL (2025-11-22 13:09:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 362.360754ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [..X.XX..XX..................................................] duration:
460.453382ms
FAIL (2025-11-22 13:09:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 460.474272ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
339.29921ms
FAIL (2025-11-22 13:09:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 339.327846ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X...X..X....................................................] duration:
371.809024ms
FAIL (2025-11-22 13:08:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 371.838508ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X...X....................................................] duration:
424.000114ms
FAIL (2025-11-22 13:08:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 424.036963ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
616.225208ms
FAIL (2025-11-22 13:08:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 616.246207ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X...X..X...................................................] duration:
383.716324ms
FAIL (2025-11-22 13:08:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 383.734188ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
433.82743ms
FAIL (2025-11-22 13:08:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 433.852258ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X...X...................................................] duration:
462.725294ms
FAIL (2025-11-22 13:08:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 462.747195ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [...X.XX..XX.................................................] duration:
580.948476ms
FAIL (2025-11-22 13:07:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 581.11058ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
602.197998ms
FAIL (2025-11-22 13:07:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 602.22047ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X...X..X..................................................] duration:
378.81664ms
FAIL (2025-11-22 13:07:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 379.012148ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
287.771266ms
FAIL (2025-11-22 13:07:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 287.789349ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
747.182918ms
FAIL (2025-11-22 13:07:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 747.203356ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration:
382.779445ms
FAIL (2025-11-22 13:07:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 382.800725ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [....X.XX..XX................................................] duration:
431.428921ms
FAIL (2025-11-22 13:07:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 431.448016ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
396.825171ms
FAIL (2025-11-22 13:06:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 396.844719ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
863.52892ms
FAIL (2025-11-22 13:06:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 863.556423ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X...X..X.................................................] duration:
344.332828ms
FAIL (2025-11-22 13:06:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 344.347865ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
309.335558ms
FAIL (2025-11-22 13:06:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 309.3576ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X...X..X................................................] duration:
376.49199ms
FAIL (2025-11-22 13:06:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 376.510836ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [.....X.XX..XX...............................................] duration:
497.300481ms
FAIL (2025-11-22 13:06:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 497.318584ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X...X.................................................] duration:
469.009938ms
FAIL (2025-11-22 13:06:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 469.036297ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
562.62335ms
FAIL (2025-11-22 13:06:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 562.644057ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
395.136732ms
FAIL (2025-11-22 13:06:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 395.165577ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X...X................................................] duration:
291.365649ms
FAIL (2025-11-22 13:05:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 291.38772ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
528.460368ms
FAIL (2025-11-22 13:05:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 528.473214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X...X..X...............................................] duration:
357.066489ms
FAIL (2025-11-22 13:05:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 357.08319ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [......X.XX..XX..............................................] duration:
463.567134ms
FAIL (2025-11-22 13:05:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 463.632847ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
293.679463ms
FAIL (2025-11-22 13:05:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 293.703467ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X...X...............................................] duration:
434.603016ms
FAIL (2025-11-22 13:05:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 434.622673ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration:
327.791569ms
FAIL (2025-11-22 13:04:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 327.819953ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
546.753484ms
FAIL (2025-11-22 13:04:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 546.774953ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X...X..X..............................................] duration:
383.642456ms
FAIL (2025-11-22 13:04:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 383.658014ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [.......X.XX..XX.............................................] duration:
437.617896ms
FAIL (2025-11-22 13:04:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 437.635078ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
555.808324ms
FAIL (2025-11-22 13:04:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 555.848068ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
282.754462ms
FAIL (2025-11-22 13:04:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 282.775161ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [........X.XX..XX............................................] duration:
473.622663ms
FAIL (2025-11-22 13:03:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 473.642158ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X...X..X.............................................] duration:
353.306916ms
FAIL (2025-11-22 13:03:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 353.462579ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration:
503.877523ms
FAIL (2025-11-22 13:03:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:30:39.886919105 (625h17m6.587864s) [upload=549895]: Starting upload
Nov 22 12:30:39.886980411 (625h17m6.587925306s) [upload=549895]: Done starting upload
Nov 22 12:30:39.887019784 (625h17m6.587964679s) [upload=549895]: Uploading segments...
Nov 22 12:30:39.887191426 (625h17m6.588136321s) [upload=549895]: Got next segment. Inline: false
Nov 22 12:30:39.887235859 (625h17m6.588180745s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:39.887239496 (625h17m6.588184392s) [upload=549895]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:39.908569558 (625h17m6.609514453s) [seg_pos={0 0}, upload=549895]: Begin upload segment...
Nov 22 12:30:39.908628519 (625h17m6.609573404s) [seg_pos={0 0}, upload=549895]: Done begin upload segment.
Nov 22 12:30:39.908652163 (625h17m6.609597058s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.908706776 (625h17m6.60965169s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.909410276 (625h17m6.610355211s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910210787 (625h17m6.611155702s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.910932301 (625h17m6.611877216s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.911590336 (625h17m6.612535261s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913261722 (625h17m6.614206617s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.913998844 (625h17m6.61494376s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.914323395 (625h17m6.61526829s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915319312 (625h17m6.616264207s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.915699105 (625h17m6.61664402s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.916318698 (625h17m6.617263593s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.917305910 (625h17m6.618250795s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918017736 (625h17m6.618962651s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.918371419 (625h17m6.619316304s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:39.919354294 (625h17m6.620299189s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Uploading piece...
Nov 22 12:30:40.023989509 (625h17m6.724934404s) [upload=549895]: Next returned nil segment
Nov 22 12:30:40.024025717 (625h17m6.724970602s) [upload=549895]: Waiting for error group managing segments...
Nov 22 12:30:40.036250214 (625h17m6.737195099s) [noise=true , address=145.79.227.11:4730 , piece_id=2LINYYXHYPSAFM4IGHBVBMDP5XF65U5HA5ATI6FK7KW7EINSUWVA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036563442 (625h17m6.737508327s) [noise=true , address=145.79.227.32:4834 , piece_id=FLD325B2ARNCKAV6YLXZGCA77ZNUXSIJQAQTV2EQB2463QZWKNJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.036837225 (625h17m6.7377821s) [noise=true , address=145.79.227.24:4583 , piece_id=STMTQY3CXU2SMQ4SZ6HKT3BDQSR56GZWPIUFTAY7OROASITQPGKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037099617 (625h17m6.738044533s) [noise=true , address=145.79.227.4:4187 , piece_id=WRFXI7KIAMXSCHIJEH2SCOKDV4SLDEW4SIHXQLRIS4CZXEW7MWIA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037192601 (625h17m6.738137517s) [noise=true , address=145.79.227.17:4970 , piece_id=ZPUIDVCML3HWFLEZS556ZT77ZDHAEU4PS3MXVJSFHOQUKSAUMNGQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037375424 (625h17m6.738320309s) [noise=true , address=145.79.227.21:4475 , piece_id=XR7GUSXOZM6J7JTSGTUYWBDC7PTK2CTLAB774KTMU6MUP3UT467Q, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037636545 (625h17m6.73858143s) [noise=true , address=145.79.227.23:4546 , piece_id=4G5DW2PW5SM27TUG73C7KB2BBJFQC57YOEAFPKHFRLTF6O2JKKEA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.037902083 (625h17m6.738846968s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4PNGOAEFU24IVKRWQ7X6HG232CFVPXBUPVWEKGRJ2O5D5SEDKA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038075589 (625h17m6.739020504s) [noise=true , address=145.79.227.14:4840 , piece_id=23EKOYDCOOO5EPSNY5NE66FQGWXOAPYGJK3NKDQHONWQCL42YGIQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038193680 (625h17m6.739138555s) [noise=true , address=145.79.227.27:4691 , piece_id=MHWK4LC4VUDCRLKA6H7WYWAYRLZ6CFVA5LZZY6OYW2ZCPMKQRKNA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.038734154 (625h17m6.739679068s) [noise=true , address=145.79.227.20:4438 , piece_id=SWI3CLSPCEAQCMMFKNLWYTAUVEOXBEQ4OCGMI53ZRHGPFWQVOZJA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:30:40.039831352 (625h17m6.740776237s) [noise=true , address=145.79.227.16:4912 , piece_id=SQGGF5O5OIVGXDONIGY7JF3NPP7VF5567WYDK5DOKN3NZ55B5VLA, seg_pos={0 0}, upload=549895]: Done uploading piece. err: <nil>
Nov 22 12:31:09.871391115 (625h17m36.57233603s) [noise=true , address=77.90.31.17:4772 , piece_id=GMGFW33CB4F23QKDGUOYFEG7QD5IBIR4SO5DFKI4LZMPOMX4HOOQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 5.161.89.252:48132->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871482266 (625h17m36.572427181s) [noise=true , address=77.90.31.21:4878 , piece_id=ET4Q4RVLH3C54GE3KN52BQBYONZVNPXUPBWD4JPEPQBHPNM5AVCQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:09.871505240 (625h17m36.572450155s) [noise=true , address=77.90.31.14:4662 , piece_id=OTS5BHNGAH3RAJIPEP7IHGCGMAMKPVUH7GK4JW2BVF7HZIO73GFQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871583757 (625h17m36.572528683s) [noise=true , address=77.90.31.19:4843 , piece_id=QWCEERFGV6L2QYRX7I7J4AXTY6ZPPBAR73MLBDJPN4M4TJFKVZEQ, seg_pos={0 0}, upload=549895]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 5.161.89.252:13858->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:09.871617520 (625h17m36.572562436s) [seg_pos={0 0}, upload=549895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:09.890558149 (625h17m36.591503064s) [upload=549895]: Done uploading segments...
Nov 22 12:31:09.890575301 (625h17m36.591520216s) [upload=549895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 503.90296ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_us_qa_orbiter_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
269.255634ms
FAIL (2025-11-22 13:03:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
16.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.756660214 (625h19m42.457605109s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757173867 (625h19m42.458118762s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757626676 (625h19m42.458571561s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.757985049 (625h19m42.458929944s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.758530111 (625h19m42.459475006s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.759740352 (625h19m42.460685247s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.760912421 (625h19m42.461857316s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.762827194 (625h19m42.463772079s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.765837693 (625h19m42.466782588s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Uploading piece...
Nov 22 12:33:15.832676241 (625h19m42.533621136s) [upload=549931]: Next returned nil segment
Nov 22 12:33:15.832689135 (625h19m42.53363403s) [upload=549931]: Waiting for error group managing segments...
Nov 22 12:33:15.856414501 (625h19m42.557359396s) [noise=true , address=145.79.227.12:4780 , piece_id=PRLVQOOWEGI74HTLUPHPWRN67WA32H6K2HYAITXJOG7FAYCWBJEA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:33:53.787018154 (625h20m20.487963049s) [noise=true , address=77.90.31.21:4886 , piece_id=CQL5XXCYI3TWVIHMSROS2C2TEZXB6EXYK737KILGJ36LGREEHINA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:14.013176308 (625h20m40.714121203s) [noise=true , address=77.90.31.14:4673 , piece_id=W7ATE5HNODO2WJYWZSDKBNNBNVH3ZSUL4SYDHMSAIKQLQTMXKQ4Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: <nil>
Nov 22 12:34:15.711282503 (625h20m42.412227398s) [noise=true , address=45.91.116.11:4129 , piece_id=UMG2YYF2YAIOO466OZMIF6PF5J3JE7PVBQGHON6Y5CYMZS4JKT5Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1G1vKtMNbawZnTDutUimTNy3L1UG75hzEvKrgJEs1rXS8Z956X, address:45.91.116.11:4129): protocol: rpc: dial tcp 45.91.116.11:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711385015 (625h20m42.412329901s) [noise=true , address=77.90.31.17:4790 , piece_id=2K66KEVZWLBOE62DFEANAWCYNO6VL554S5GCCDQ7GODOCVSVI3HA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711462481 (625h20m42.412407366s) [noise=true , address=45.91.116.5:4306 , piece_id=ZUEFM5MQ26LZQQ4C5KWGGEYEKMYCCI7S6X6EDAC245PSTDYLDQWQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711472209 (625h20m42.412417094s) [noise=true , address=77.90.31.19:4868 , piece_id=YZUNACONDKRO6PFK7TSZOFAMOOJAJDCSZVFFLGVNJG7Z2M3VT2UA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711543283 (625h20m42.412488178s) [noise=true , address=45.91.116.14:4194 , piece_id=DSJKXDMRWHOKOYWBHVCPYEHV7BL5AL24MWI44QG6XNPTNCFK752A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711562628 (625h20m42.412507514s) [noise=true , address=77.90.31.12:4599 , piece_id=XRK7RNWSOVHQKIYCPITG5VHGOKCN3MQ3ETWSVX5ZBQRM2MI2EMDA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: write tcp 5.161.89.252:14146->77.90.31.12:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711623903 (625h20m42.412568798s) [noise=true , address=77.90.31.18:4835 , piece_id=2PYGYS5JT7WCKTJF7FZB6I2TKD32W7NUOJWQYDOVYYJLNL2CVTTA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711647608 (625h20m42.412592503s) [noise=true , address=45.91.116.10:4093 , piece_id=EE6XK7MLGUKHCMAKZDTIXIBGLUXKAKGSDJJWS6FME4IOTDHAUD2Q, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: rpc: dial tcp 45.91.116.10:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711719082 (625h20m42.412663977s) [noise=true , address=77.90.31.10:4542 , piece_id=JH26ZP5PZQ3IEAYL2SXPXVXAZLHKVQUWTPFU65GSI5JBJDS44IMA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711727117 (625h20m42.412672012s) [noise=true , address=45.91.116.13:4180 , piece_id=7OGQMFSVBLM4UAZ52LFM3GGKNVLJEXTVBBQQV34QWV7OO4KRA5BA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: rpc: dial tcp 45.91.116.13:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711754368 (625h20m42.412699264s) [noise=true , address=45.91.116.4:4967 , piece_id=67AWYR5X2AMLGIRVDNCZLYOODGCJJQ2YCPE2LU7KNV7S4BCNUQAA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711806406 (625h20m42.412751291s) [noise=true , address=45.91.116.6:4992 , piece_id=2PDM6KUYF2CDP5KSFDH2ZHP6LHV5375TL4F2NHQCG2J7EM3OFGJQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx, address:45.91.116.6:4992): protocol: write tcp 5.161.89.252:9490->45.91.116.6:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711883540 (625h20m42.412828426s) [noise=true , address=45.91.116.9:4279 , piece_id=5L3RHTINCC7NAVGCEWIIBYZP4LSAFO242PTYKGOQERMD55KK6WFQ, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711904931 (625h20m42.412849816s) [noise=true , address=77.90.31.11:4557 , piece_id=D5KFZPKLJPEDXRRM6YIDY4VXIS24Y2SI43377K7ZYTKOR4RUGV6A, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 5.161.89.252:45754->77.90.31.11:4557: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.711991243 (625h20m42.412936138s) [noise=true , address=45.91.116.7:4036 , piece_id=4FX2XDR5ZPMVLONHEPG2XQQ6BPAN3PCVAZDRKDMUROS3O72PWJSA, seg_pos={0 0}, upload=549931]: Done uploading piece. err: ecclient: upload failed (node:1svXoJFMPUuwBW6UsMpRAzCrxRuQWQbLkZGEcfePd7GR8bTTm5, address:45.91.116.7:4036): protocol: rpc: dial tcp 45.91.116.7:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:15.712010449 (625h20m42.412955344s) [seg_pos={0 0}, upload=549931]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Nov 22 12:34:15.727287843 (625h20m42.428232759s) [upload=549931]: Done uploading segments...
Nov 22 12:34:15.727306879 (625h20m42.428251795s) [upload=549931]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 269.275552ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_us_qa_orbiter_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=3) [.........X.XX..XX...........................................] duration:
461.884459ms
FAIL (2025-11-22 13:03:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 461.905568ms
core
uplink
us-select-1
us1-placement-20
|