Stock-SEC-Data-Dashboard - GitLite

KO | EN

gitlite — search

#react #typescript #android #machine-learning #rust #deep-learning #linux #docker #awesome #cli #vue #pytorch #openai #open-source

Stock-SEC-Data-Dashboard

(★ 109)

This project aims to generate an ETL pipeline for SEC EDGAR system data dashboard and analysis.

파일 탐색기

최종 버전 다운로드 (.zip)

- - sec_data_pipeline.py
- - config-template.conf
  - redshift_to_csv.py
  - s3_to_redshift.py
  - sec_data_etl.py
  - upload_data_s3.py
- docker-compose.yaml
- schema.yml
- sec_transformed.sql
- aws-setup.md
- config.md
- dbt-run-output.png
- dbt.md
- dbt_transformed_out.png
- docker-airflow.md
- infra-setup.md
- pipeline.png
- starter.md
- visuals.md
- edgar_eda_nb.ipynb.zip
- main.tf
- output.tf
- variable.tf
.gitignore
LICENSE
README.md

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/hrshtsharma17/Stock-SEC-Data-Dashboard

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd Stock-SEC-Data-Dashboard

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Docker

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.

docker compose -f airflow/docker-compose.yaml up -d --build

compose 설정 파일에 정의된 서비스들을 대상으로 명령을 실행합니다.

✅ 터미널에 docker compose ps 를 입력해 컨테이너들이 Up 상태인지 확인하세요. README에 포트 번호가 적혀있다면 브라우저에서 http://localhost:포트번호 로 접속해보세요.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SEC Data Dashboard Project

## Overview

The Stock SEC Data Dashboard project is designed not only to create a dashboard but also to serve as a learning experience, introducing users to a range of tools, helping them develop new skills, and potentially assisting others in similar data-related endeavors. This versatile tool can be adapted to work with various forms of data, such as interactions on social networks, financial trading data, common crawl data, and more.

![The Data Pipeline](https://raw.githubusercontent.com/hrshtsharma17/Stock-SEC-Data-Dashboard/master/documents/pipeline.png "Data Extraction Pipeline")

## Project Workflow

The project workflow is divided into several key steps:

1. **AWS Infrastructure Setup**: This step involves configuring Amazon Web Services (AWS) resources, including Amazon S3 for object storage and Amazon Redshift for data warehousing. We use Terraform for infrastructure-as-code to quickly set up these resources.

2. **Configuration**: A configuration file (`configuration.conf`) is created to store project-specific details like AWS credentials, Redshift connection information, and other configurations required for data extraction and loading.

3. **Docker & Airflow Setup**: Apache Airflow is used to orchestrate the pipeline. Docker is employed to create and manage containers, simplifying the setup process. This step ensures that the necessary containers and services are running to facilitate data extraction, transformation, and loading.

4. **Data Extraction and Loading**: Within the Airflow environment, a DAG (Directed Acyclic Graph) named `sec_data_pipeline` is defined. This DAG automates the extraction of data from SEC, uploading it to an S3 bucket, and loading it into Amazon Redshift. Tasks in this DAG utilize the `BashOperator` to run Python scripts for data processing.

5. **Data Transformation with dbt (Optional)**: This optional step involves using dbt to connect to the data warehouse and perform data transformations. While not a core part of the project, it offers an opportunity to explore dbt and build skills in data transformation.

6. **Visualization with BI Tools (Optional)**: The project also allows for connecting Business Intelligence (BI) tools to the data warehouse to create visualizations. Google Data Studio is recommended, but users are free to choose other BI tools based on their preferences and requirements.

## Getting Started

To begin using the Stock SEC Data Dashboard project, follow these steps:

For more detailed steps visit the [project starter](documents/starter.md)

1. **AWS Setup**: Create a personal AWS account if you don't already have one. Secure your account by enabling multi-factor authentication (MFA) and set up an IAM user for admin permissions. Configure AWS CLI with your credentials.

2. **Infrastructure Setup**: Use Terraform to set up AWS resources for S3, Redshift, and related components. Customize the configuration in `variables.tf` to match your project requirements.

3. **Configuration**: Create a `configuration.conf` file in the `airflow/pipeline` directory to store project-specific details such as AWS configurations, Redshift credentials, and bucket names.

4. **Docker & Airflow Setup**: Install Docker and Docker Compose on your machine. Initialize the Airflow environment using Docker Compose, and ensure that the necessary containers are up and running. Access the Airflow Web Interface at `http://localhost:8080` and monitor the `sec_data_pipeline` DAG.

5. **Data Extraction and Loading**: The `sec_data_pipeline` DAG runs automatically to extract data from sec, upload it to S3, and load it into Redshift. Monitor the DAG's status in the Airflow UI.

6. **Optional Steps**: Explore optional steps such as data transformation with dbt and visualization with BI tools to enhance your project further.

## Project Details

- **Project Structure**: The project's structure includes directories for infrastructure (Terraform), configuration (AWS and Airflow), data extraction (Python scripts), and optional steps like dbt and BI tools integration.

- **Customization**: Feel free to customize the project by modifying configurations, adding new data sources, or integrating additional tools as needed.

## Contributors

- Harshit Sharma

## License

This project is licensed under the [MIT License](LICENSE).