KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #codex #typescript #cli #deepseek-harness #claude-code #ai-agents #mcp #rust

spark_app_twitter

★ 87 Open GitHub ↗

A data engineering project (Twitter monitor app)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# spark_app_twitter
## Monitoring Twitter Trends

App for monitoring trends on Twitter, we use the Ukranian War (2022) as a case scenario,
the project can be used for different purposes just modifying the `config.json` file in `/tweepy_ingestion`, and also the `functions.py` 
in `/tweepy_ingestion/utilities` in order to change the topics that will be created automatically.

### Architecture schema:

![title](https://raw.githubusercontent.com/jmcmt87/spark_app_twitter/main/screenshots/Untitled-Diagram-2-2-8-drawio-2.png)

```

### Requirementes

* Twitter Developer Account
* Twitter API v2 Bearer Token
* AWS S3 Credentials
* AWS bucket
* MongoDB Atlas cluster

### Start:

* 1st step: launching the kafka cluster, for that it's necessary to have docker-compose installed, launch it from
`/kafka_cluster_step` using the command:

```
docker-compose up -d
```
* 2nd step: launching the script `/tweepy_ingestion/main.py` to start ingesting data and sending it to the kafka cluster.

* 3rd step: launching the spark application to start processing the data `/spark_app/main.py`.

* 4th step: launching the streamlit application in `/streamlit`, for that it's needed to use the command:

```
streamlit run main.py
```

You can use every Pipfile and Pipfile.lock to recreate the **pipenv environment** in the different directories:

```
pipenv sync

pipenv shell

python3 main.py
```

# Screenshots

![title](https://raw.githubusercontent.com/jmcmt87/spark_app_twitter/main/screenshots/screenshot_app.png)

![title](https://raw.githubusercontent.com/jmcmt87/spark_app_twitter/main/screenshots/screenshot_app_2.png)

![title](https://raw.githubusercontent.com/jmcmt87/spark_app_twitter/main/screenshots/screenshot_app_3.png)

spark_app_twitter

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README