dfp_external_storage

KO | EN

gitlite — search

#javascript #react #typescript #llm #linux #macos #nodejs #java #vue #awesome-list #pytorch #security #openai #open-source

dfp_external_storage

(★ 59)

Simplest cloud file management for Frappe / ERPNext. S3 compatible external bucket can be assigned per Frappe folder, allowing you to fine-tune the location of your Frappe / ERPNext "File"s: within local filesystem or to exteral S3 bucket.

dfp_external_storage Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DFP External Storage

> **😊 Thanks!**
>
> If you find this code useful, please help me (https://github.com/sponsors/developmentforpeople) to keep it updated, improved and safe. Thank you very very much for your help 🫶!

Simplest cloud file management for Frappe / ERPNext. S3 compatible external bucket can be assigned per Frappe folder, allowing you to fine-tune the location of your Frappe / ERPNext "File"s: within local filesystem or to exteral S3 bucket.

[![Frappe files within S3 buckets](https://github.com/developmentforpeople/dfp_external_storage/assets/47140294/e762a286-b2c7-4a9b-a7e3-486b9a0892d6)](https://www.youtube.com/embed/2uTnWZxhtug)

## Examples / Use cases

### All Frappe / ERPNext files into external S3 compatible bucket

[upload_frappe_erpnext_files_s3_compatible_bucket.webm](https://github.com/developmentforpeople/dfp_external_storage/assets/47140294/68592d26-4391-45fc-bd75-d4d5f06ce899)

### Move files / objects from S3 compatible bucket to another S3 compatible bucket (between buckets in same or different connection)

[move_objects_from_one_s3_compatible_bucket_to_another.webm](https://github.com/developmentforpeople/dfp_external_storage/assets/47140294/9c4d7197-d19e-422e-85a9-8af7725014f0)

### Move files / objects from S3 compatible bucket to local file system

[move_objects_from_s3_compatible_to_local_filesystem.webm](https://github.com/developmentforpeople/dfp_external_storage/assets/47140294/2d4eccf1-f7e2-4c89-9694-95ec36b6856d)

### Move files in local filesystem to S3 compatible bucket

[move_local_files_to_s3_compatible_bucket.webm](https://github.com/developmentforpeople/dfp_external_storage/assets/47140294/6a19d3b6-48c6-46a1-a08d-29d3555b4419)

### Per file examples

[move_file_from_s3_compatible_bucket_to_different_one_then_to_local_file.webm](https://github.com/developmentforpeople/dfp_external_storage/assets/47140294/1a4f216a-a6b4-4728-a27e-efdf4cbcf983)

### List all remote files in bucket

Shows all files in bucket, even the ones not in Frappe File doctype.

[list_files_in_remote_s3_bucket.webm](https://github.com/developmentforpeople/dfp_external_storage/assets/47140294/fbd38418-686e-45b4-b23b-048bed4d1143)

### Customizable

Choose the best setup for you: S3 only for all site files or specified folders, use S3 / Minio presigned urls, cache or not small files, etc.

![Settings](https://github.com/developmentforpeople/dfp_external_storage/assets/47140294/0ad2f24b-d37d-4882-80c4-c1e77a74f666)

## Requirements

- Frappe version >= 14 (Try to use last version: 15)

## Functionalities

- S3 bucket can be defined per folder/s. If "Home" folder defined, all Frappe / ERPNext files will use that S3 bucket.
- Files accesible with custom URLs: /file/[File ID]/[file name.extension]
- Frappe / ERPNext private/public functionality is preserved for external files. If an external private file is loaded not having access a not found page will be showed.
- External Storages can be write disabled, but files will be visible yet.
- Bulk file relocation (upload and download). You can filter by local S3 bucket/local filesystem and then change all those files to a different S3 bucket or to local filesystem. All files are "moved" without load them fully in memory optimizing large ones transfer.
- Small icon allows you visualize if file is within an S3 bucket.
- Same file upload (same file hash) will reuse existent S3 key and is not reuploaded. Same functionality as Frappe has with local files.
- Choosed S3 bucket file listing tool.
- S3 bucket can not be deleted if has "File"s assigned / within it.
- If bucket is not accesible file will be uploaded to local filesystem.
- Stream data in chunks to and from S3 without reading whole files into memory (thanks to [Khoran](https://github.com/khoran))
- List all remote objects in bucket (includes too the ones not uploaded trough Frappe)
- Support for S3 / Minio presigned urls: allowing video streaming capabilities and other S3 functionalities.
- Presigned url can be used for all files in defined folders but defined by mimetype.
- Files are now streamed by default.
- Extended settings per External Storage doc:
  - Cache only files smaller than
  - Cache for x seconds
  - Stream buffer size
  - Presigned url activation
  - Presigned url only for mimetypes defined
  - Presigned url expiration
  - Use S3 file size instead of saved on Frappe File (needed for files > 2GB)
- ... maybe I am forgetting something ;)

### Flow options

- No S3 external storages defined
- or S3 external storages defined but not assigned to folders:
  - All uploaded files are saved in local filesystem
- One S3 external storage assigned to "Attachments" folder:
  - Only files uploaded to that folder will be use that S3 bucket
- One S3 external storage assigned to "Home" folder:
  - All files uploaded to Frappe will be located within that bucket. Except the files uploaded to "Attachments" that will use the above defined bucket

### File actions available

- If a "File" has an "DFP External Storage" assigned.
  - If changed to a different "DFP External Storage" file will be:
    - "downloaded" from previous bucket > "uploaded" to new bucket > "deleted" from previous bucket.
  - If leaved empty, file will be "downloaded" to local filesystem > "deleted" from bucket.
- If a "File" has no "DFP External Storage" assigned, so it is in local filesystem:
  - If assigned a "DFP External Storage", file will be:
    - "uploaded" to that bucket > "deleted" from filesystem

## Setup or try it locally

### Install Frappe 15

Follow all steps for your OS within official guide: [https://docs.frappe.io/framework/v15/user/en/installation](https://docs.frappe.io/framework/v15/user/en/installation).

### Create your personal "frappe-bench" environment (customizable folder name)

Into your home folder:

```
cd ~
bench init frappe-bench
```

### Install "dfp_external_storage" app

```
cd ~/frappe-bench
# Version 15: recommended
bench get-app --branch version-15 git@github.com:developmentforpeople/dfp_external_storage.git
# Version 14: deprecated
bench get-app --branch version-14 git@github.com:developmentforpeople/dfp_external_storage.git
# Not stable but with latest features (for testing or developing)
bench get-app --branch develop git@github.com:developmentforpeople/dfp_external_storage.git
```

### Create a new site with "dfp_external_storage" app installed on it

```
cd ~/frappe-bench
bench new-site site_name.localhost --install-app dfp_external_storage

# If you need to delete site to recreate it (Be careful! all data will be lost, only for testing!):
bench drop-site --no-backup --force site_name.localhost
```

### Initialize servers to get site running

```
cd ~/frappe-bench
bench start
```

### Create one or more "DFP External Storage"s

Add one or more S3 bucket and, this is the most important step, assign "Home" folder to it. This makes all files uploaded to Frappe / ERPNext being uploaded to that bucket.

You can select a different folder and only those files will be uploaded, or select different buckets for different folders, your imagination is your limit!! :D

### Stream data to and from S3 without reading whole files into memory

Option is valuable when working with large files.

For uploading content from a local file, usage would look like:

```
file_doc = frappe.get_doc({
    "doctype":"File",
    "is_private":True,
    "file_name": "file name here"
})
file_doc.dfp_external_storage_upload_file(filepath)
file_doc.save()
```

To download content to a local file:

```
file_doc = frappe.get_doc("File",doc_name)
file_doc.dfp-external_storage_download_to_file("/path/to/local/file")
```

To read remote file directly via a proxy object:

```
file_doc = frappe.get_doc("File",doc_name)

#read zip file table of contents without downloading the whole zip file
with zipfile.ZipFile(file_doc.dfp_external_storage_file_proxy()) as z:
  for zipinfo in z.infolist():
     print(f"{zipinfo.filename}")

```

### Uninstall "dfp_external_storage" app

- [Frappe Docs: Uninstall app](https://docs.frappe.io/framework/v15/user/en/bench/reference/uninstall-app)
- [Frappe Docs: Bench cheatsheet](https://docs.frappe.io/framework/v15/user/en/bench/resources/bench-commands-cheatsheet)

```bash
# First uninstall app from site:
bench --site {site} uninstall-app {app} --no-backup
# Then remove app from bench:
bench remove-app {app}
```

## Pending

- Make tests:
  - Create DFP External Storage
  - Upload file to bucket
  - Read bucket file
  - Relocate bucket file
  - Delete bucket file

## Contributing

1. [Code of Conduct](CODE_OF_CONDUCT.md)

## Attributions

- cloud storage by Iconstock from [Noun Project](https://thenounproject.com/browse/icons/term/cloud-storage/)

## License

MIT

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=developmentforpeople/dfp_external_storage&type=date&legend=bottom-right)](https://www.star-history.com/#developmentforpeople/dfp_external_storage&type=date&legend=bottom-right)

dfp_external_storage

dfp_external_storage Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

dfp_external_storage

dfp_external_storage Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README