cs-insights

KO | EN

gitlite — search

#hacktoberfest #react #go #rust #deep-learning #macos #nodejs #windows #awesome #pytorch #security #ios #mcp #open-source

(★ 13)

The main controller for services in the cs-insights project through docker-compose.

cs-insights Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div id="top"></div>

  <br/>
  <div align="center">
  <a href="https://cs-insights.uni-goettingen.de">
    <img src="images/logo-with-text.png" alt="Logo" width="500">
  </a>
  <br/>
  <br/>
  <a href="https://jpwahle.github.io/cs-insights/"><strong>Explore the docs »</strong></a>
  <br />
  <br />
  <a href="http://cs-insights.uni-goettingen.de/">View Demo</a>
  ·
  <a href="https://github.com/users/jpwahle/projects/1">Project Board</a>
  ·
  <a href="https://jpwahle.github.io/cs-insights-uptime/">Uptime Tracker</a>
  ·
  <a href="https://github.com/jpwahle/cs-insights/issues/new?assignees=&labels=&template=bug_report.md&title=">Report Bug</a>
  ·
  <a href="https://github.com/jpwahle/cs-insights/issues/new?assignees=&labels=&template=feature_request.md&title=">Request Feature</a>
  </div>
<br />
<div align="center">
<table>
  <tr>
  <td><a href="https://github.com/jpwahle/cs-insights-frontend"><img src="https://img.shields.io/badge/GitHub-Frontend-6466e9?style=for-the-badge"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-frontend/issues"><img src="https://img.shields.io/github/issues/jpwahle/cs-insights-frontend.svg?style=for-the-badge&color=389fe3"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-frontend/graphs/contributors"><img src="https://img.shields.io/github/contributors/jpwahle/cs-insights-frontend.svg?style=for-the-badge&color=32c1cb"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-frontend/stargazers"><img src="https://img.shields.io/github/stars/jpwahle/cs-insights-frontend.svg?style=for-the-badge&color=2dd4bf"/></a></td>
  </tr>
  <tr>
  <td><a href="https://github.com/jpwahle/cs-insights-backend"><img src="https://img.shields.io/badge/GitHub-Backend-6466e9?style=for-the-badge"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-backend/issues"><img src="https://img.shields.io/github/issues/jpwahle/cs-insights-backend.svg?style=for-the-badge&color=389fe3"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-backend/graphs/contributors"><img src="https://img.shields.io/github/contributors/jpwahle/cs-insights-backend.svg?style=for-the-badge&color=32c1cb"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-backend/stargazers"><img src="https://img.shields.io/github/stars/jpwahle/cs-insights-backend.svg?style=for-the-badge&color=2dd4bf"/></a></td>
  </tr>
  <tr>
  <td><a href="https://github.com/jpwahle/cs-insights-crawler"><img src="https://img.shields.io/badge/GitHub-Crawler-6466e9?style=for-the-badge"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-crawler/issues"><img src="https://img.shields.io/github/issues/jpwahle/cs-insights-crawler.svg?style=for-the-badge&color=389fe3"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-crawler/graphs/contributors"><img src="https://img.shields.io/github/contributors/jpwahle/cs-insights-crawler.svg?style=for-the-badge&color=32c1cb"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-crawler/stargazers"><img src="https://img.shields.io/github/stars/jpwahle/cs-insights-crawler.svg?style=for-the-badge&color=2dd4bf"/></a></td>
  </tr>
  <tr>
  <td><a href="https://github.com/jpwahle/cs-insights-prediction-endpoint"><img src="https://img.shields.io/badge/GitHub-Prediction-6466e9?style=for-the-badge"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-prediction-endpoint/issues"><img src="https://img.shields.io/github/issues/jpwahle/cs-insights-prediction-endpoint.svg?style=for-the-badge&color=389fe3"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-prediction-endpoint/graphs/contributors"><img src="https://img.shields.io/github/contributors/jpwahle/cs-insights-prediction-endpoint.svg?style=for-the-badge&color=32c1cb"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-prediction-endpoint/stargazers"><img src="https://img.shields.io/github/stars/jpwahle/cs-insights-backend.svg?style=for-the-badge&color=2dd4bf"/></a></td>
  </tr>
  <tr>
  <td><a href="https://github.com/jpwahle/cs-insights-uptime"><img src="https://img.shields.io/badge/GitHub-Uptime-6466e9?style=for-the-badge"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-uptime/issues"><img src="https://img.shields.io/github/issues/jpwahle/cs-insights-uptime.svg?style=for-the-badge&color=389fe3"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-uptime/graphs/contributors"><img src="https://img.shields.io/github/contributors/jpwahle/cs-insights-uptime.svg?style=for-the-badge&color=32c1cb"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-uptime/stargazers"><img src="https://img.shields.io/github/stars/jpwahle/cs-insights-backend.svg?style=for-the-badge&color=2dd4bf"/></a></td>
  </tr>
  <tr>
  <td><a href="https://github.com/jpwahle/cs-insights-main"><img src="https://img.shields.io/badge/GitHub-main-6466e9?style=for-the-badge"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-main/issues"><img src="https://img.shields.io/github/issues/jpwahle/cs-insights-main.svg?style=for-the-badge&color=389fe3"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-main/graphs/contributors"><img src="https://img.shields.io/github/contributors/jpwahle/cs-insights-main.svg?style=for-the-badge&color=32c1cb"/></a></td>
  <td><a href="https://github.com/jpwahle/cs-insights-main/stargazers"><img src="https://img.shields.io/github/stars/jpwahle/cs-insights-backend.svg?style=for-the-badge&color=2dd4bf"/></a></td>
  </tr>
</table>

<br/>
<br/>
<strong>Demo Credentials </strong><br/>
user: demo@cs-insights.com <br/>
password: demo
</div>

## About The Project

![An overview of the components in the CS-Insights ecosystem (in GitHub light mode)](https://raw.githubusercontent.com/jpwahle/cs-insights/main/images/system-overview-light.png)
![An overview of the components in the CS-Insights ecosystem (in GitHub dark mode)](https://raw.githubusercontent.com/jpwahle/cs-insights/main/images/system-overview-dark.png)

CS-Insights is an open-source project that makes information about computer science research available with the click of a button.
We use the [DBLP Discovery Dataset D3](https://bit.ly/d3-dataset) to show trends in volume, topics, impact, and many more.
The [frontend](https://github.com/jpwahle/cs-insights-frontend) enables abstract access to the data through visualizations and filters. The [backend](https://github.com/jpwahle/cs-insights-backend) enables access to the data through a well-defined REST API and makes it easy to index new data as well. The [prediction-endpoint](https://github.com/jpwahle/cs-insights-prediction-endpoint) does the heavy lifting for analyzing topics and other semantic features using parents and childrens of docker containers that can run on different servers. The [crawler](https://github.com/jpwahle/cs-insights-crawler) makes sure that the latest data is retrieved from the web, parsed, and stored into the backend. Finally, the [uptime](https://github.com/jpwahle/cs-insights-uptime) tracker periodically queries all endpoints to check their status and response times.

### Built With

[![React][react.js]][react-url]
[![Node][passport]][passport-url]
[![Node][swagger]][swagger-url]
[![Node][express]][express-url]
[![Node][node.js]][node-url]
[![Node][mongodb]][mongo-url]
[![Node][docker]][docker-url]

## Getting Started

To get a local copy up and running follow these simple example steps. The project relies on a singe docker-compose.yml file that is used to develop and deploy all the services.

### Prerequisites

You need to have `docker` and the `docker-compose-plugin` installed.

### Installation

First, clone the repository _and its submodules_ locally.

```
git clone --recurse-submodules -j5 https://github.com/gipplab/cs-insights-main.git
```

If you are developing, it makes sense to start with the submodules (e.g., frontend, backend, ...) at their main branch. To update all submodules to main, run:

```
git submodule foreach git pull origin main
```

To start the development environment, run the following command:

```sh
docker compose --env-file .env.development -f docker-compose.dev.yml up --build
```

This will start each service (e.g., backend, frontend) in development mode with hot reload. Whenever you change any files of the sub-repositories (e.g., ./cs-insights-frontend/src/App.tsx) the development server will reload and show the new rendered frontend.

To run the production environment in detached mode, run the following command:

```sh
docker compose up -d
```

<details>
  <summary>HTTPS Certificate Renewal</summary>
  
  SSL/TLS certificates are valid for 90 days. To renew then automatically install a cronjob that checks the renewal weekly on sundays at 9:00 UTC which is the time that fewest users are up to see the downtime. 
  
  ```sh
  sudo crontab -e
  ```
  
  ```sh
  00 9 * * 0 certbot renew --dry-run --pre-hook "docker compose -f /home/jp/cs-insights-main/docker-compose.yml stop frontend" --post-hook "docker compose -f /home/jp/cs-insights-main/docker-compose.yml start frontend"
  ```
</details>

<details>
  <summary>Database Updates</summary>
  
  To update the database with the latest release, run the following commands (replace admin/admin_password with the production secrets):
  
  ```sh
  mongoimport --db csinsights --collection papers --file papers.jsonl --upsertFields=corpusid --mode=upsert --authenticationDatabase=admin --username admin --password admin_password --numInsertionWorkers 12
  
  mongoimport --db csinsights --collection authors --file authors.jsonl --upsertFields=authorid --mode=upsert --authenticationDatabase=admin --username admin --password admin_password --numInsertionWorkers 12
  ```

This will update all existing entries and add new entries on the respective id fields.

</details>

<details>
  <summary>Production Secrets</summary>
  
  Some secret variables should only be available encrypted as environment variables in the production environment.
  Therefore, the docker-compose.yml contains docker secrets encrypted on the host server. To export the secrets on the server, run the following command:
  
  ```sh
  docker swarm init
  ```
  Or you can also join a swarm using

```sh
docker swarm join --token <token> <manager-ip>:2377
```

where manager-ip is cs-insights.uni-goettingen.de.

Then create the secrets of the docker-compose-yml with the following command:

```sh
printf "<secret>" | docker secret create <secret_name> -
```

Alternatively to exporting external secrets and referring to them with

```
mongo_password:
  external: true
```

you can also store them in text files on the host system and give docker-compose the path

```
mongo_password:
  file: mongo_password.txt
```

Supporting the secrets in files is compatible with docker-compose, while external secrets rely on docker stack deploy

```
docker stack deploy -c docker-compose.yml cs-insights
```

</details>

## Usage

The services are available on the following ports:

- frontend: `80/443` or `3001` (dev)
- backend: `80/443` or `3000` (dev)
- prediction-endpoint: `8000`

_For more examples, please refer to the individual API documentation for [backend](https://jpwahle.github.io/cs-insights-backend/), [frontend](https://jpwahle.github.io/cs-insights-frontend/), [prediction-endpoint](https://jpwahle.github.io/cs-insights-prediction-endpoint/), and [crawler](https://jpwahle.github.io/cs-insights-crawler/)_.

## Features

For more details on the features of CS-Insights, please refer to our paper [here](https://arxiv.org/abs/2210.06878).
 
<p align="right">(<a href="#top">back to top</a>)</p>
  
  

  
## Roadmap

See the [project board](https://github.com/users/jpwahle/projects/1) for a full list of planned features (and known issues).

## Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
6. We will get in touch and include your feature in the next release.

## License

Distributed under the MIT License. See `LICENSE` for more information.

## Contact

Jan Philip Wahle - [@jpwahle](https://twitter.com/jpwahle) - wahle@gipplab.org
Terry Ruas - [@ruasterry](https://twitter.com/ruasterry) - ruas@gipplab.org

Project Link: [https://github.com/jpwahle/cs-insights](https://github.com/jpwahle/cs-insights)

## Citation

```
@misc{Ruas2022,
  doi = {10.48550/ARXIV.2210.06878},
  url = {https://arxiv.org/abs/2210.06878},
  author = {Ruas, Terry and Wahle, Jan Philip and Küll, Lennart and Mohammad, Saif M. and Gipp, Bela},
  keywords = {Computation and Language (cs.CL), Digital Libraries (cs.DL), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {CS-Insights: A System for Analyzing Computer Science Research},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution Share Alike 4.0 International}
}
```

[react.js]: https://img.shields.io/badge/React-20232A?style=for-the-badge&logo=react&logoColor=61DAFB
[react-url]: https://reactjs.org/
[node.js]: https://img.shields.io/badge/Node.js-43853D?style=for-the-badge&logo=node.js&logoColor=white
[node-url]: https://nodejs.org/en/
[mongodb]: https://img.shields.io/badge/MongoDB-4EA94B?style=for-the-badge&logo=mongodb&logoColor=white
[mongo-url]: https://www.mongodb.com/
[docker]: https://img.shields.io/static/v1?style=for-the-badge&message=Docker&color=2496ED&logo=Docker&logoColor=FFFFFF&label=
[docker-url]: https://www.docker.com/
[passport]: https://img.shields.io/static/v1?style=for-the-badge&message=Passport&color=222222&logo=Passport&logoColor=34E27A&label=
[passport-url]: https://www.passportjs.org/
[swagger]: https://img.shields.io/static/v1?style=for-the-badge&message=Swagger&color=222222&logo=Swagger&logoColor=85EA2D&label=
[swagger-url]: https://swagger.io/
[express]: https://img.shields.io/static/v1?style=for-the-badge&message=Express&color=000000&logo=Express&logoColor=FFFFFF&label=
[express-url]: https://expressjs.com/

cs-insights

cs-insights Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

cs-insights

cs-insights Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README