s3pypi - GitLite

KO | EN

gitlite — search

#javascript #ai #typescript #llm #machine-learning #deep-learning #linux #docker #macos #vue #ios #openai #mcp #open-source

s3pypi

(★ 353)

CLI tool for creating a Python Package Repository in an S3 bucket.

파일 탐색기

최종 버전 다운로드 (.zip)

- - tests.yml
- handler.py
- put_user.py
- test_handler.py
- __init__.py
- __main__.py
- core.py
- exceptions.py
- index.py
- locking.py
- storage.py
- migrate-s3-index.py
- main.tf
- - conftest.py
  - test_locking.py
  - test_main.py
  - test_storage.py
- - test_core.py
  - test_index.py
- __init__.py
- conftest.py
.gitignore
CHANGELOG.md
CONTRIBUTING.md
LICENSE
Makefile
poetry.lock
pyproject.toml
README.md
setup.cfg
tox.ini

🚀 설치 가이드

1. 코드 내려받기

git clone https://github.com/gorilla-co/s3pypi

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd s3pypi

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

$ pip install s3pypi

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

$ pip install your-project --extra-index-url https://pypi.example.com/PREFIX/

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

3. Make

보통

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Make Linux/macOS는 보통 기본 설치되어 있습니다. Windows는 별도 설치(예: MSYS2, WSL)가 필요합니다.

make

생성된 빌드 설정을 바탕으로 실제 컴파일을 진행해 실행 파일을 만듭니다.

✅ 에러 없이 끝나면 성공입니다. 생성된 실행 파일을 직접 실행해보세요.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# S3PyPI

S3PyPI is a CLI for creating a Python Package Repository in an S3 bucket.

## Why?

The official [Python Package Index (PyPI)](https://pypi.org) is a public
repository of Python software. It's used by `pip` to download packages.

If you work at a company, you may wish to publish your packages somewhere
private instead, and still have them be accessible via `pip install`. This
requires hosting your own repository.

S3PyPI enables hosting a private repository at a low cost. It requires only an
[S3 bucket] for storage, and some way to serve files over HTTPS (e.g. [Amazon
CloudFront]).

Publishing packages and index pages to S3 is done using the `s3pypi` CLI.
Creating the S3 bucket and CloudFront distribution is done using a provided
[Terraform] configuration, which you can tailor to your own needs.

[S3 bucket]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
[Amazon CloudFront]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html
[Terraform]: https://www.terraform.io/

## Alternatives

- [AWS CodeArtifact](https://aws.amazon.com/codeartifact/) is a fully managed
  service that integrates with IAM.

## Getting started

### Installation

Install s3pypi using pip:

```console
$ pip install s3pypi
```

### Setting up S3 and CloudFront

Before you can start using `s3pypi`, you must set up an S3 bucket for storing
packages, and a CloudFront distribution for serving files over HTTPS. Both of
these can be created using the [Terraform] configuration provided in the
`terraform/` directory:

```console
$ git clone https://github.com/gorilla-co/s3pypi.git
$ cd s3pypi/terraform/

$ terraform init
$ terraform apply
```

You will be asked to enter your desired AWS region, S3 bucket name, and domain
name for CloudFront. You can also enter these in a file named
`config.auto.tfvars`:

```terraform
region = "eu-west-1"
bucket = "example-bucket"
domain = "pypi.example.com"
```

#### DNS and HTTPS

The Terraform configuration assumes that a [Route 53 hosted zone] exists for
your domain, with a matching (wildcard) certificate in [AWS Certificate
Manager]. If your certificate is a wildcard certificate, add
`use_wildcard_certificate = true` to `config.auto.tfvars`.

#### Distributed locking with DynamoDB

To ensure that concurrent invocations of `s3pypi` do not overwrite each other's
changes, the objects in S3 can be locked via an optional DynamoDB table (using
the `--lock-indexes` option). To create this table, add `enable_dynamodb_locking
= true` to `config.auto.tfvars`.

#### Basic authentication

To enable basic authentication, add `enable_basic_auth = true` to
`config.auto.tfvars`. This will attach a [Lambda@Edge] function to your
CloudFront distribution that reads user passwords from [AWS Systems Manager
Parameter Store]. Users and passwords can be configured using the `put_user.py`
script:

```console
$ basic_auth/put_user.py pypi.example.com alice
Password:
```

This creates a parameter named `/s3pypi/pypi.example.com/users/alice`. Passwords
are hashed with a random salt, and stored as JSON objects:

```json
{
  "password_hash": "7364151acc6229ec1468f54986a7614a8b215c26",
  "password_salt": "RRoCSRzvYJ1xRra2TWzhqS70wn84Sb_ElKxpl49o3Y0"
}
```

#### Terraform module

The Terraform configuration can also be included in your own project as a
module:

```terraform
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

module "s3pypi" {
  source = "github.com/gorilla-co/s3pypi//terraform/modules/s3pypi"

bucket = "example-bucket"
  domain = "pypi.example.com"

use_wildcard_certificate = true
  enable_dynamodb_locking  = true
  enable_basic_auth        = true

providers = {
    aws.us_east_1 = aws.us_east_1
  }
}
```

#### AWS WAF

AWS WAF rules and rate limits tend to be very specific to each deployment, so
this project does not try to standardize a particular configuration. Instead,
the module accepts the ARN of an existing [AWS WAFv2 Web ACL] via the
`waf_web_acl_arn` variable, and associates it with the CloudFront
distribution. The Web ACL must have scope `CLOUDFRONT` and be created in the
`us-east-1` region:

```terraform
resource "aws_wafv2_web_acl" "pypi" {
  provider = aws.us_east_1
  name     = "pypi"
  scope    = "CLOUDFRONT"

default_action {
    allow {}
  }

rule {
    name     = "AWSManagedRulesCommonRuleSet"
    priority = 0

override_action {
      none {}
    }

statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesCommonRuleSet"
        vendor_name = "AWS"
      }
    }

visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "AWSManagedRulesCommonRuleSet"
      sampled_requests_enabled   = true
    }
  }

visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "pypi"
    sampled_requests_enabled   = true
  }
}
```

Then pass its ARN to the module: `waf_web_acl_arn = aws_wafv2_web_acl.pypi.arn`.
If you need the distribution's ID or ARN for other purposes (e.g. Shield
Advanced, CloudWatch alarms), they are exposed as the
`cloudfront_distribution_id` and `cloudfront_distribution_arn` outputs.

[AWS WAFv2 Web ACL]: https://docs.aws.amazon.com/waf/latest/developerguide/web-acl.html

#### Migrating from s3pypi 0.x to 1.x

Existing resources created using the CloudFormation templates from s3pypi 0.x
can be [imported into Terraform] and [removed from CloudFormation]. For example:

```console
$ terraform init
$ terraform import module.s3pypi.aws_s3_bucket.pypi example-bucket
$ terraform import module.s3pypi.aws_cloudfront_distribution.cdn EDFDVBD6EXAMPLE
$ terraform apply
```

[imported into Terraform]: https://www.terraform.io/docs/import/index.html
[removed from CloudFormation]: https://aws.amazon.com/premiumsupport/knowledge-center/delete-cf-stack-retain-resources/

In this new configuration, CloudFront uses the S3 REST API endpoint as its
origin, not the S3 website endpoint. This allows the bucket to remain private,
with CloudFront accessing it through an [Origin Access Identity (OAI)]. To make
this work with your existing S3 bucket, all `<package>/index.html` objects must
be renamed to `<package>/`. You can do so using the provided script:

```console
$ scripts/migrate-s3-index.py example-bucket
```

To instead keep using the old configuration with a publicly accessible S3
website endpoint, pass the following options when uploading packages:

```console
$ s3pypi upload ... --index.html --s3-put-args='ACL=public-read'
```

[Origin Access Identity (OAI)]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html

### Example IAM policy

The `s3pypi` CLI requires the following IAM permissions to access S3 and
(optionally) DynamoDB. Replace `example-bucket` by your S3 bucket name.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::example-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::example-bucket"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:*:*:table/example-bucket-locks"
    }
  ]
}
```

## Usage

### Distributing packages

You can now use `s3pypi` to upload packages to S3:

```console
$ cd /path/to/your-project/
$ python setup.py sdist bdist_wheel

$ s3pypi upload dist/* --bucket example-bucket [--prefix PREFIX]
```

See `s3pypi --help` for a description of all options.

### Installing packages

Install your packages using `pip` by pointing the `--extra-index-url` to your
CloudFront domain. If you used `--prefix` while uploading, then add the prefix
here as well:

```console
$ pip install your-project --extra-index-url https://pypi.example.com/PREFIX/
```

Alternatively, you can configure the index URL in `~/.pip/pip.conf`:

```
[global]
extra-index-url = https://pypi.example.com/PREFIX/
```

## Roadmap

Currently there are no plans to add new features to s3pypi. If you have any
ideas for new features, check out our [contributing guidelines](CONTRIBUTING.md)
on how to get these on our roadmap.

## Contact

Got any questions or ideas? We'd love to hear from you. Check out our
[contributing guidelines](CONTRIBUTING.md) for ways to offer feedback and
contribute.

## License

Copyright (c) [Gorillini NV](https://gorilla.co).
All rights reserved.

Licensed under the [MIT](LICENSE) License.

[Route 53 hosted zone]: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/AboutHZWorkingWith.html
[AWS Certificate Manager]: https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html
[Lambda@Edge]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html
[AWS Systems Manager Parameter Store]: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html