KO
|
EN
gitlite — search
Search
#javascript
#python
#hacktoberfest
#react
#ai
#typescript
#llm
#go
#golang
#android
#machine-learning
#rust
#deep-learning
#linux
landgriffon
★ 19
Open GitHub ↗
No description available.
Download README (.md)
Explore Similar Repositories
linkauto
:
Simple to use wrapper around the private LinkedIn API, written in async Python.
keyframes_tween
:
A tween that allow defining keyframes for various properties.
saferollout
:
No description available.
LEO-HO-Project
:
Handover Self-Optimization in Drones Based on Low Earth Orbit Satellite Systems Project Codes
colab_automation
:
Automating Google Colab with JavaScript to run prescheduled and dynamic Python scripts
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
landgriffon
?
Download (.md)
# LandGriffon <!-- omit from toc --> Homepage: [github.com/Vizzuality/landgriffon](https://github.com/Vizzuality/landgriffon) --- - [1. Project description](#1-project-description) - [2. Architecture](#2-architecture) - [3. Folder Structure](#3-folder-structure) - [4. Environment variables](#4-environment-variables) - [4.1. How to use environment variables in local development](#41-how-to-use-environment-variables-in-local-development) - [4.2. Environment variables \& secrects in Github Actions](#42-environment-variables--secrects-in-github-actions) - [4.3. Environment variables \& secrets in Terraform](#43-environment-variables--secrets-in-terraform) - [5. Services](#5-services) - [5.1. API Service](#51-api-service) - [5.2. Client Service](#52-client-service) - [5.3. Marketing site](#53-marketing-site) - [5.4. Tiler server (TiTiler)](#54-tiler-server-titiler) - [5.5. Database server (Postgresql)](#55-database-server-postgresql) - [5.6. Cache and Message Broker server (Redis)](#56-cache-and-message-broker-server-redis) - [6. Utilities](#6-utilities) - [6.1. Data package](#61-data-package) - [6.2. Infrastructure package](#62-infrastructure-package) - [7. Start Up](#7-start-up) - [7.1. System requirements](#71-system-requirements) - [7.2. Data initialization](#72-data-initialization) - [7.3. Configure environment variables](#73-configure-environment-variables) - [7.4. Starting all the services](#74-starting-all-the-services) - [8. Local development](#8-local-development) - [8.1. Using docker with debugging and hot reload](#81-using-docker-with-debugging-and-hot-reload) - [8.1.1. Only frontend](#811-only-frontend) - [8.1.2. Only backend](#812-only-backend) - [8.2. Debugging with hot reload manually](#82-debugging-with-hot-reload-manually) - [8.2.1. Only frontend](#821-only-frontend) - [8.2.2. Only backend](#822-only-backend) - [9. Testing](#9-testing) - [10. CI/CD and Deployment](#10-cicd-and-deployment) - [10.1. GitHub Actions](#101-github-actions) - [10.1.1. Testing workflows](#1011-testing-workflows) - [10.1.2. Deployment Workflows](#1012-deployment-workflows) - [10.1.4. Manual triggering](#1014-manual-triggering) - [10.2. Pull request template](#102-pull-request-template) - [10.3. Cloud infrastructure (Terraform)](#103-cloud-infrastructure-terraform) - [11. Additional documentation?](#11-additional-documentation) - [12. TODO: Documentation/application improvements](#12-todo-documentationapplication-improvements) ## 1. Project description Deforestation and water stress have a negative impact on agricultural supply chains, preventing agribusiness and food companies from becoming more sustainable. Advanced technology such as the [Copernicus programme](https://dataspace.copernicus.eu/) provides precise, timely and easily accessible data that improve environmental management and mitigate climate change effects. The EU-funded LAND GRIFFON project will develop digital decision-making instruments based on [Copernicus data](https://dataspace.copernicus.eu/) to observe, prognoses, analyse and follow environmental impacts on the entire agricultural supply chain. These innovative instruments will support agribusiness and food enterprises in becoming more sustainable and transparent. **Related Information**: - [Executive summary](https://landgriffon.com/docs/LG_Methodology_Executive_Summary.pdf) with an overview of how LandGriffon works. - [Full methodology](https://landgriffon.com/docs/LG_Methodology_Technical_Note.pdf) with an in-depth description of every feature. > TODO: User Guide would also be useful as the [Full methodology](https://landgriffon.com/docs/LG_Methodology_Technical_Note.pdf) document is not exactly a user guide, only explains how the indicators work and other related information, which does not exactly fit as a user guide. ## 2. Architecture This repository is a [monorepo](https://monorepo.tools/#understanding-monorepos) containing all the microservices of the LandGriffon platform, each organized in a top-level folder. All services are packaged as [Docker](https://www.docker.com/) images, ensuring consistency and ease of deployment. For local development, the microservices run seamlessly using [Docker Compose](https://docs.docker.com/compose/). In CI, testing, staging, and production environments, the microservices are orchestrated via [Kubernetes](https://kubernetes.io/). > TODO: Add a diagram showing how all components relate to each other. ## 3. Folder Structure - `.` (**root**): Root folder with configuration, documentation, startup ([Docker Compose](https://docs.docker.com/compose/)) and environment variable files. - [`.editorconfig`](./.editorconfig): defines consistent formatting rules for different file types. - [`.gitignore`](./.gitignore): excludes files from version control. - [`.pre-commit-config.yaml`](./.pre-commit-config.yaml): configures pre-commit hooks to automate code quality checks and formatting before commits. - [`.python-version`](./.python-version): specifies the Python version to use for the LandGriffon project. - [`CHANGELOG.md`](./CHANGELOG.md): (TODO: outdated) track important changes made to LandGriffon over time. - [`docker-compose.yml`](./docker-compose.yml): (TODO: improve) [Docker Compose](https://docs.docker.com/compose/) file to start all services in local environment. - [`ENV_VARS.md`](./ENV_VARS.md): Document explaining the environment variables needed by service in LandGriffon. - [`env.default`](./env.default): Template with placeholders and default values for the environment variables needed for the several services of LandGriffon. - [`LICENSE`](./LICENSE): License file. - [`Makefile`](./Makefile): Starting point for several processes. (deprecated?) - `README.md`: this document. - [`.github/`](./.github): stores GitHub-specific configuration files for testing, publishing, building and deploying workflows. Also contains a pull request template. - [`api/`](./api/): contains the API service code. - [`client/`](./client/): contains the client (frontend) service code. - `cookie-traceability/`: (TODO: deprecated? remove?) - [`data/`](./data/): contains the data and scripts needed to initialize the database. - [`database/`](./database/): contains a `Dockerfile` to build a custom Postgresql image. - [`infrastructure/`](./infrastructure/): Terraform files for automatic cloud provisioning. - `landing/`: (TODO: deprecated? remove?) - [`marketing/`](./marketing/): current static site for LandGriffon - [`redis/`](./redis/): contains a `Dockerfile` to build a custom Redis image. - [`tiler/`](./tiler/): necessary files to start up a Tiler service with [TiTiler](https://developmentseed.org/titiler/) including a `Dockerfile`. ## 4. Environment variables > TODO: Not all services are included in the [ENV_VARS.md](./ENV_VARS.md), only the API. You can find all the environment variables needed for every service in the [ENV_VARS.md](./ENV_VARS.md) file. ### 4.1. How to use environment variables in local development > TODO: It would be nice to have this method in place. > > - Create a `.env` file at the root of the repository out of the [`env.default`](./env.default) template. > - Default values are already included. > - User-provided values need to go in the environment variables that do not have value. You can download these values from the **shared secret storage**. Choose the ones for local development. ### 4.2. Environment variables & secrects in Github Actions > TODO: Explain how are they used ### 4.3. Environment variables & secrets in Terraform > TODO: Explain how are they used ## 5. Services The services that make up LandGriffon are: - API service ([Nest.js](https://nestjs.com/)) - Client service ([Next.js](https://nextjs.org/)) - Marketing site ([Next.js](https://nextjs.org/)) - Tiler server ([TiTiler](https://developmentseed.org/titiler/)) - Database server ([Postgresql](https://www.postgresql.org/)) - Cache and queue server ([Redis](https://redis.io/)) ### 5.1. API Service The API provides endpoints that the frontend utilizes to retrieve all necessary information and execute the calculations essential for Landgriffon's proper functionality. It ensures seamless communication between the frontend and backend, handling data processing, computations, and any required transformations to support the application's features effectively. Its contents are in the [`api/`](./api/) folder. More information its [README.md](./api/README.md) file. ### 5.2. Client Service Web frontend client for the API. Its contents, including a Dockerfile, are in the [`client/`](./client/) folder. More information its [README.md](./client/README.md) file. ### 5.3. Marketing site The marketing service serves as the current static site for LandGriffon. The live marketing site can be accessed at [landgriffon.com](https://landgriffon.com/). Its contents are in the [`marketing/`](./marketing/) folder. In the [Methodology](https://landgriffon.com/methodology) section of the marketing site, you'll find two key documents: one providing an overview of how LandGriffon works and another offering a detailed explanation of its features. - [Executive summary](https://landgriffon.com/docs/LG_Methodology_Executive_Summary.pdf) with an overview of how LandGriffon works. - [Full methodology](https://landgriffon.com/docs/LG_Methodology_Technical_Note.pdf) with an in-depth description of every feature. More information in its [README.md](./marketing/README.md) file. ### 5.4. Tiler server (TiTiler) Tiler service that uses [TiTiler](https://developmentseed.org/titiler/), _"a modern dynamic tile server built on top of FastAPI and [Rasterio](https://rasterio.readthedocs.io/en/stable/)/[GDAL](https://gdal.org/en/stable/)"_. Its contents, including a [`Dockerfile`](./tiler/Dockerfile), are in the [`tiler/`](./tiler/) folder. More information in its [README.md](./tiler/README.md) file. ### 5.5. Database server (Postgresql) Database service containing the relational data needed for LandGriffon. It uses [PostgreSQL](https://www.postgresql.org/) v14 with two additional plugins: - [PostGIS](https://postgis.net/) v3.5.2 - [PostgreSQL bindings for H3](https://github.com/bytesandbrains/h3-pg) v3.7.2 The [`Dockerfile`](./database/Dockerfile) and the [`entrypoint.sh`](./database/entrypoint.sh) script are in the [`database/`](./database/) folder. More information in this [link](./api/README.md#31-database-setup). ### 5.6. Cache and Message Broker server (Redis) Redis serves as both a message broker and a caching mechanism for certain precomputed results. While caching can be disabled through configuration environment variables, Redis is essential for message broker functionality and cannot be turned off. More information about the use cases where Redis is needed in LandGriffon are explained in its [README.md](./redis/README.md) file. ## 6. Utilities ### 6.1. Data package This directory contains seed data and scripts required to import essential data into the database, ensuring that the API has access to the necessary information for proper operation. These scripts automate the data population process, facilitating a smooth setup and maintenance of the database while supporting the API’s functionality. Its contents are in the [`data/`](./data/) folder. More information about the data process used in LandGriffon are explained in the following [README.md](./data/README.md) file. ### 6.2. Infrastructure package The code related to setting up the cloud infrastructure is located in the [`infrastructure/`](./infrastructure/) folder. This setup adheres to the principle of [Infrastructure as Code](https://en.wikipedia.org/wiki/Infrastructure_as_code) (IaC), which allows for the management and provisioning of computing resources through machine-readable configuration files. By using IaC, we ensure that our infrastructure is consistent, repeatable, and version-controlled. We utilize [Terraform](https://www.terraform.io/), a powerful IaC tool, to define and provision our infrastructure on [AWS](https://aws.amazon.com/). Terraform enables us to create, update, and manage AWS resources efficiently and reliably, ensuring that our infrastructure is scalable and maintainable. For more details on how to deploy and manage the infrastructure, please refer to this [README.md](./infrastructure/README.md) file. ## 7. Start Up To start the application for the first time you have to import some data into the database, starting just the database and running the data seed process. > TODO: review if this is the real way of working. Once you already have the data imported, you just can start the whole system by running: ```sh docker compose up ``` ### 7.1. System requirements > TODO: Complete with more dependencies? - [Python](https://www.python.org/) - [Node.js](https://nodejs.org/en) - [Docker](https://www.docker.com/) ### 7.2. Data initialization In order to run the application, it first needs to start a database and ingest the needed data. This is a time consuming process (a one time process for local development) that needs to be run separately, before starting the application. How to setup the database is explained in this [`README.md`](./database/README.md) file. Also, for detailed information on how to import the data, read this [`README.md`](./data/README.md) file. ### 7.3. Configure environment variables > TODO: Explain: > > - [ ] which environment variables > - [ ] templates with default values suitable for local development (`env.default`) > - [ ] where to get values for the different environment variables with _"real"_ credentials for external services > (Geocoding, email, ...) > - [ ] explain where the _real_ credentials are stored _per environment_ (staging, production, ...) > - [ ] explain how the application uses the environment variables (maybe at service level in its corresponding > `README.md` files) ### 7.4. Starting all the services > TODO: This should be ideal, not sure if it works with a proper configuration of the environment variables. After all the previous steps you just start the whole application: ```sh docker compose up ``` ## 8. Local development > TODO: Explain the different possibilities to start specific services for development maybe with some environment > variables we can tweak the ports exposed by the containers to be usable for debugging and hot reloading inside the container. ### 8.1. Using docker with debugging and hot reload #### 8.1.1. Only frontend #### 8.1.2. Only backend ### 8.2. Debugging with hot reload manually #### 8.2.1. Only frontend #### 8.2.2. Only backend ## 9. Testing > TODO: Explain the different ways of testing. > > SUGGESTION: In a monorepo setup and using some monorepo utilities ([Turborepo](https://turbo.build/repo/docs), > [Nx](https://nx.dev/), ...) we can test all the services at the same time (frontend, backend, data?) with a single > command from the root directory, and take advantage of the caching features all this tools come with. In LandGriffon we have the following types of tests: - **Unit tests**: verifies the correctness of individual code units in isolation. - **Integration tests**: individual components or modules are combined and tested as a group to ensure that they work together correctly. - **API tests**: validate the functionality of the API to ensure they correctly handle requests, deliver expected responses, and integrate seamlessly with other components. - **E2E tests**: simulate real user scenarios to validate that all components of an application work together as intended from start to finish. For mor information on how to trigger testing in specific modules, refer to its corresponding documentation: - [Testing the backend](./api/README.md) - [Testing the frontend](./client/README.md) - [Testing the data processing](./data/README.md) ## 10. CI/CD and Deployment ### 10.1. GitHub Actions This document provides an overview of the [GitHub Actions](https://github.com/features/actions) workflows used in the LandGriffon project. #### 10.1.1. Testing workflows - **`testing-api.yml`**: Runs tests for the API service, ensuring code quality and functionality. - **`testing-client.yml`**: Executes integration tests for the client application. - **`testing-client-unit-tests.yml`**: Runs unit tests specifically for the client components. - **`testing-data-import.yml`**: Validates data import functionality and integrity. - **`testing-tiler.yml`**: Tests the tiler service for geospatial data handling. #### 10.1.2. Deployment Workflows - **`deploy-to-kubernetes.yml`**: Deploys the application to a Kubernetes cluster, typically triggered on merge to main/specific branches. - **`publish-docker-images.yml`**: Builds and pushes Docker images to a container registry. - **`publish-marketing-site.yml`**: Builds and deploys the marketing website. - **`build-database-docker.yml`**: Builds a Docker image for the PostgreSQL database with required extensions. #### 10.1.4. Manual triggering To manually trigger a workflow: 1. Navigate to the Actions tab in the repository 2. Select the desired workflow 3. Click `Run workflow` 4. Select the branch and provide any required inputs ### 10.2. Pull request template This GitHub pull request template provides a standardized format for contributors, requiring them to document changes with descriptions and testing instructions while enforcing quality standards through a pre-merge checklist. It ensures PRs include Jira ticket references, proper testing, CI validation, and code reviews, ultimately streamlining the review process and maintaining project quality standards. ### 10.3. Cloud infrastructure (Terraform) > TODO: Explain the main options taken for the deployment in the cloud: > > - [ ] AWS provider options. > - [ ] GCP provider options. > - [ ] GitHub Actions secrets. ## 11. Additional documentation? - [Executive summary](https://landgriffon.com/docs/LG_Methodology_Executive_Summary.pdf) with an overview of how LandGriffon works. - [Full methodology](https://landgriffon.com/docs/LG_Methodology_Technical_Note.pdf) with an in-depth description of every feature. > TODO: User guide. ## 12. TODO: Documentation/application improvements - **Environment variables** needed for every service are, incomplete, outdated and dispersed over several files. - **The process to setup the environment variables** for local development needs some clarification and a way to _"do-it-without-asking-anyone"_ would be a nice thing to have. A template with default values, a guide on how to create a `.env` file and a reference to the **shared secret storage** for the sensible values (passwords, users, secrets...) should be enough. - **Initial data setup** takes too much time so the possiblity to have a _"working relevant subset"_ of the data for just the minimum local development would also be nice to have.
