KO
|
EN
gitlite โ search
Search
#typescript
#ai-agents
#ai
#dsh-plugin
#deepseek-harness
#open-source
#cli
#claude-code
#codex
#developer-tools
#react
#windows
helyim
โ 209
Open GitHub โ
seaweedfs implemented in pure Rust
Download README (.md)
Explore Similar Repositories
cubo
:
On-Demand Earth System Data Cubes (ESDCs) in Python
tbparse
:
Load tensorboard event logs as pandas DataFrames for scientific plotting; Supports both PyTorch and TensorFlow
clickhouse-kafka-connect
:
ClickHouse Kafka Connector
ClassGPT
:
ChatGPT for lecture slides
What-to-eat-today
:
๐ฝ๏ธๅบไบๅพRAGๆๆฏ็AI็พ้ฃๆจ่ๅฉๆ - Datawhale all-in-ragๆ็จๅฎๆๆกไพ๏ผ้ๆNeo4jๅพๆฐๆฎๅบใMilvusๅ้ๆฃ็ดขไธๆบ่ฝๅฏน่ฏ็ณป็ป
// repository documentation
Was this content helpful?
โ 0
(0 ratings)
Select Rating:
โ
โ
โ
โ
โ
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
helyim
?
Download (.md)
<div align="center"> <h1>Helyim</h1> <h4> <a href="https://github.com/seaweedfs/seaweedfs">seaweedfs</a> implemented in pure Rust. Please โญ on <a href="https://github.com/helyim/helyim">github</a>! </h4> [](https://crates.io/crates/helyim) [](https://github.com/helyim/helyim/actions/workflows/build.yaml) [](https://codecov.io/github/helyim/helyim)  </div> ### Features #### Additional Features - [x] Can choose no replication or different replication levels, rack and data center aware. - [x] Automatic compaction to reclaim disk space after deletion or update. - [x] Automatic master servers fail over - no single point of failure (SPOF). - [x] **Erasure Coding** for warm storage Rack-Aware 10.4 erasure coding reduces storage cost. ### Installation ```shell cargo +nightly install helyim ``` ### Usage By default, the master node runs on port 9333, and the volume nodes run on port 8080. Let's start one master node, and one volume node on port 8080. Ideally, they should be started from different machines. We'll use localhost as an example. Helyim uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format. #### 1. Start Master Server ```shell cargo run --release --bin helyim master ``` #### 2. Start Volume Servers ```shell cargo run --release --bin helyim volume --port 8080 --folders ./target ``` #### 3. Write File To upload a file: first, send a HTTP POST, PUT, or GET request to `/dir/assign` to get an `fid` and a volume server URL: ```bash curl http://localhost:9333/dir/assign {"fid":"6,16b7578a5","url":"127.0.0.1:8080","public_url":"127.0.0.1:8080","count":1,"error":""} ``` Second, to store the file content, send a HTTP multipart POST request to `url + '/' + fid` from the response: ```bash curl -F file=@./sun.jpg http://127.0.0.1:8080/6,16b7578a5 {"name":"sun.jpg","size":1675569,"error":""} ``` To update, send another POST request with updated file content. For deletion, send an HTTP DELETE request to the same `url + '/' + fid` URL: ```bash curl -X DELETE http://127.0.0.1:8080/6,16b7578a5 ``` #### 4. Start Filer Server ```shell cargo run --release --bin helyim filer ``` To upload a file without assigning file id ```shell curl -F file=@./sun.jpg http://127.0.0.1:8888/sun.jpg ``` ### Failover Master Server When initiating a Raft cluster, it is necessary to specify the same node sequence when starting the Leader and Follower instances. You can view the cluster status by accessing `http://localhost:9333/cluster/status`. ```bash # start master1 cargo run --bin helyim -- master --ip 127.0.0.1 --port 9333 \ --peers 127.0.0.1:9333 \ --peers 127.0.0.1:9335 \ --peers 127.0.0.1:9337 # start master2 cargo run --bin helyim -- master --ip 127.0.0.1 --port 9335 \ --peers 127.0.0.1:9333 \ --peers 127.0.0.1:9335 \ --peers 127.0.0.1:9337 # start master3 cargo run --bin helyim -- master --ip 127.0.0.1 --port 9337 \ --peers 127.0.0.1:9333 \ --peers 127.0.0.1:9335 \ --peers 127.0.0.1:9337 ``` ### Benchmark My laptop results on Lenovo IdeaPad Pro 16 (2023) with SSD, CPU: 14 Intel Core i9 5.4GHz. It seems to be slower than `seaweedfs`, especially in terms of reading. ```console โ ./weed benchmark -server=localhost:9333 This is SeaweedFS version 0.76 linux amd64 ------------ Writing Benchmark ---------- Completed 15199 of 1048576 requests, 1.4% 15198.1/s 15.3MB/s Completed 31887 of 1048576 requests, 3.0% 16687.9/s 16.8MB/s Completed 48439 of 1048576 requests, 4.6% 16551.6/s 16.7MB/s ... Completed 994044 of 1048576 requests, 94.8% 16645.2/s 16.8MB/s Completed 1010800 of 1048576 requests, 96.4% 16755.8/s 16.9MB/s Completed 1027412 of 1048576 requests, 98.0% 16612.2/s 16.7MB/s Completed 1044319 of 1048576 requests, 99.6% 16907.0/s 17.0MB/s Concurrency Level: 16 Time taken for tests: 63.249 seconds Complete requests: 1048576 Failed requests: 0 Total transferred: 1106759553 bytes Requests per second: 16578.50 [#/sec] Transfer rate: 17088.29 [Kbytes/sec] Connection Times (ms) min avg max std Total: 0.1 0.9 29.8 0.4 Percentage of the requests served within a certain time (ms) 50% 0.9 ms 66% 1.0 ms 75% 1.1 ms 90% 1.3 ms 95% 1.5 ms 98% 1.7 ms 99% 1.8 ms 100% 29.8 ms ------------ Randomly Reading Benchmark ---------- Completed 89963 of 1048576 requests, 8.6% 89957.6/s 90.5MB/s Completed 187560 of 1048576 requests, 17.9% 97597.1/s 98.2MB/s Completed 283486 of 1048576 requests, 27.0% 95925.8/s 96.6MB/s Completed 382035 of 1048576 requests, 36.4% 98549.4/s 99.2MB/s Completed 480649 of 1048576 requests, 45.8% 98613.9/s 99.3MB/s Completed 583585 of 1048576 requests, 55.7% 102933.7/s 103.6MB/s Completed 683954 of 1048576 requests, 65.2% 100370.9/s 101.0MB/s Completed 782522 of 1048576 requests, 74.6% 98567.9/s 99.2MB/s Completed 883504 of 1048576 requests, 84.3% 100982.7/s 101.7MB/s Completed 987320 of 1048576 requests, 94.2% 103814.3/s 104.5MB/s Concurrency Level: 16 Time taken for tests: 10.600 seconds Complete requests: 1048576 Failed requests: 0 Total transferred: 1106777459 bytes Requests per second: 98925.73 [#/sec] Transfer rate: 101969.36 [Kbytes/sec] Connection Times (ms) min avg max std Total: 0.0 0.1 2.3 0.1 Percentage of the requests served within a certain time (ms) 50% 0.1 ms 95% 0.2 ms 98% 0.4 ms 100% 2.3 ms ``` ### Acknowledgments - [seaweedfs](https://github.com/seaweedfs/seaweedfs) - SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. - [zergling](https://github.com/july2993/zergling) - seaweedFS re-implemented in Rust.
