KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

tg-keyword-trends

★ 63 Open GitHub ↗

This script searches messages containing specified search terms in Telegram channels the user is a member of.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Telegram Keyword Trends
An analysis tool to explore the emergence of hatespeech, disinformation, and narratives of interest on the Telegram chat platform. Please use this tool with caution as it does not have content moderation or filtering. You are responsible for the content that may be exported.

In short, this tool allows you to search all the channels you follow with a list of keywords/phrases and returns all matching results in various formats with graph visualisations. It also optionally downloads the media and thus can be used as a media search engine (currently some bugs with this feature - do not use as exhaustive media search tool).

###### Screenshot of tool in action, exploring hate speech during the Russian full-scale invasion of Ukraine
[![Screenshot of tool in action](https://user-images.githubusercontent.com/118008765/230943146-8c7fc77f-0b2f-4bf3-8f07-9e3d959ca30c.png "Screenshot of tool in action")](https://user-images.githubusercontent.com/118008765/230943146-8c7fc77f-0b2f-4bf3-8f07-9e3d959ca30c.png "Screenshot of tool in action")

##### Key Features
- This tool is designed to work with sockpuppets that follow many channels covering a particular topic.
- You can change your API details to use different accounts by editing the **.env** file.
- The tool is designed to work like Google Trends showing daily volume of key terms and map over time.
- Date filtering allows you to narrow a search into a shorter time period. If left blank, it automatically scales to the maximum range of the data.
- The tool uses Telegram search which means it is particularly good for Russian language searches and generally handles word endings well.
- Generates individual graphs for each key term or grouped term set.
- Generates aggregated graph showing all key terms or term groups in a search on the same graph for comparison.
- Generates percentage and rolling-percentage graphs to help compare keyword matches against estimated channel activity.
- Generates an optional wordcloud image from matching message text when the optional `wordcloud` package is installed.
- Compiles a report PDF that shows the graphs and prints the full code for auditing of data and validation of evidence.
- Outputs a TXT file summary including all the main stats, e.g, date run, channels searched, and relative volume per channel.
- Optional concurrent media download for results (this massively (like really massively) prolongs the time needed to run the tool).
- Downloaded media has filename channelid_postid so it is easy to find the original, and a JSONL manifest is used to skip duplicate downloads.

This script searches messages containing specified search terms in Telegram channels the user is a member of. It exports the search results in HTML and CSV formats, generates a report, and plots the message count per day.

It is designed to monitor trends of search terms in much the same way that Google Trends does. This can be very useful for identifying the emergence of hatespeech or discussion/narratives following certain events.

This current version does not do any significant adjustment to the data, for example, the graph does not display incidence of terms adjusted to the incidence of all messages. This means further analysis should be conducted to ensure that a sharp spike in terms is not confounded by a sharp spike in general activity. For this reason, the graph output should be treated as indicative of need for further research and statistical analysis.

###### Example result exploring hate speech during the Russian full-scale invasion of Ukraine
[![Example result exploring hate speech during the Russian full-scale invasion of Ukraine](https://user-images.githubusercontent.com/118008765/230750727-0a4f74db-9ab2-41df-b49a-c1ec2c785753.png "Example result exploring hate speech during the Russian full-scale invasion of Ukraine")](https://user-images.githubusercontent.com/118008765/230750727-0a4f74db-9ab2-41df-b49a-c1ec2c785753.png "Example result exploring hate speech during the Russian full-scale invasion of Ukraine")
*This image is an example result showing how the channels under investigation saw a surge in usage of specific terms.*

###### Example of the report generated 
[![Example of the report generated](https://user-images.githubusercontent.com/118008765/231264336-74be2122-dcec-4146-ac51-a5062a79e436.png "Example of the report generated")](https://user-images.githubusercontent.com/118008765/231264336-74be2122-dcec-4146-ac51-a5062a79e436.png "Example of the report generated")
*This image is an example result from the report, a PDF document that outlines the code and prints the script at the end. This means that no matter what changes or what version of the script is being used, the exact process can be scrutinised.*

###### Screenshot of some of the information generated in the txt stats report

[![Screenshot of some of the information generated in the report](https://user-images.githubusercontent.com/118008765/230942324-d42d96da-8df4-4a87-8201-360852b2f662.png "xxx")](https://user-images.githubusercontent.com/118008765/230942324-d42d96da-8df4-4a87-8201-360852b2f662.png "xxx")

This tool has been tested on English and Russian language search terms.

**WARNING:** This tool uses your list of followed groups as the list it searches from. It may include personal chats/groups. For the sake of OPSEC, it is recommended to use a burner account and follow only investigation-specific chats.

# Installation
Clone the tg-keyword-trends repository by running the following command in your terminal or command prompt:

``` git clone https://github.com/thomasjjj/tg-keyword-trends.git```

Navigate into the tg-keyword-trends directory:

```cd tg-keyword-trends```

Install the required Python dependencies using pip:

```pip install -r requirements.txt```

Run the tool from the repository root:

```python main.py```

# Features
- Graph adjusts scale to oldest and newest posts.
- CSV generated for further processing.
- HTML file generated for opening links.
- Generates report documenting the key details of the scrape (date, channels accessed, etc) for auditability of findings.
- Media download with duplicate tracking
- Custom channel lists
- Grouped search terms
- Percentage graphs and optional wordcloud output

# Usage:

1. Add the search terms into a .txt file. Use one term per line, or use `Group label: term | translation | transliteration` to plot several related terms as one line.
2. Make sure you have your Telegram API details ready [https://my.telegram.org/auth]. On first run, the script saves the API ID, API hash, phone number, and optional 2FA password to **.env**.
3. Choose whether to search all followed channels or provide a custom channel list. Custom channel files accept usernames, `t.me` links, `t.me/c/...` links, and numeric IDs.
4. Enter an optional date range in `dd/mm/yyyy` format. Invalid ranges are rejected and prompted again.
5. Choose whether to download matching media. Media defaults to **TG-Media/** unless `MEDIA_OUTPUT_DIR` is set in **.env**.
6. The search results will be exported as HTML, CSV, and JSON files in a timestamped output folder.
7. The script will generate a TXT report and a PDF report containing generated graphs.

# Telegram Authentication:

The script checks **.env** first for Telegram credentials. If required values are missing, it prompts for them and writes them to **.env** for future runs.

Telethon also keeps a local session file so future runs should not ask for a login code again unless that session file is deleted, expired, or invalidated.

Supported **.env** keys:

- TELEGRAM_API_ID
- TELEGRAM_API_HASH
- TELEGRAM_PHONE
- TELEGRAM_2FA_PASSWORD
- TELEGRAM_SESSION
- MEDIA_OUTPUT_DIR
- MEDIA_DOWNLOAD_CONCURRENCY

If your Telegram account has two-factor authentication enabled, the script prompts for the password in plaintext so it works in terminals that do not support hidden password prompts. That password is saved in **.env** as plaintext. Keep **.env** private and do not commit it.

Existing **api_values.txt** API credentials are migrated into **.env** automatically when **.env** does not already contain them.

# Media Downloads:

Downloaded media is saved to **TG-Media/** by default. Set `MEDIA_OUTPUT_DIR` in **.env** to use a different folder, and set `MEDIA_DOWNLOAD_CONCURRENCY` to control concurrent downloads. The default concurrency is `3`.

The media manifest is stored as `media_manifest.jsonl` in the media output folder. Each record stores the channel ID, message ID, saved path, search metadata, and source link. Existing manifest records are used to skip duplicate downloads, unless the recorded file is missing.

# Project Structure:

- **main.py**: Thin entry point for running the tool from the repository root.
- **src/tg_keyword_trends/app.py**: Main application workflow.
- **src/tg_keyword_trends/auth.py** and **env.py**: Telegram authentication and `.env` credential handling.
- **src/tg_keyword_trends/channels.py**: Followed-channel and custom-channel selection helpers.
- **src/tg_keyword_trends/files.py**: File dialogs, search-term loading, output directory creation, and HTML link rendering.
- **src/tg_keyword_trends/inputs.py**: Date, search-term group, and channel-list parsing helpers.
- **src/tg_keyword_trends/media.py**: Concurrent media downloads and manifest duplicate tracking.
- **src/tg_keyword_trends/plotting.py** and **reports.py**: Graph, wordcloud, PDF, and text report generation.
- **tests/**: Unit tests for import-safe helper modules.

# Tips:
- Due to the date filtering feature, this tool also works well as a Telegram search engine that allows date-filtered results. Simply run the search in the date window needed and open up the output html file for a list of messages that match and their links.
- The tool handles timezones automatically and adjusts for them. Be particularly careful when editing any section of the code relating to time and date formats as this was difficult to debug. 
- It is recommended that you create a dedicated Telegram account for each subject matter. This will allow you to target only relevant channels and removes noise. 
- You don't need to search singular and plural nouns separately as this is handled by Telegram's search, (generally speaking, for English and Russian language).

[![Demonstration of graph](https://user-images.githubusercontent.com/118008765/232030941-aa506853-48ba-4433-8abf-1ee454ea1e5b.png "Demonstration of graph")](https://user-images.githubusercontent.com/118008765/232030941-aa506853-48ba-4433-8abf-1ee454ea1e5b.png "Demonstration of graph")
*This image shows the useage of the various placenames for "Bakmut", including the old Soviet names. One use of this tool could be for validating the search terms used in OSINT research. As can be seen here, one may limit their collection potential if they only use the official current name for the city rather than past and controversial names too. *

# Dependencies:

- pandas~=3.0.3
- matplotlib~=3.11.0
- Telethon~=1.44.0
- colorama~=0.4.6
- Pillow~=12.2.0
- reportlab~=5.0.0
- numpy~=2.5.0
- pytz~=2026.2
- tqdm~=4.68.3

Optional:

- wordcloud: enables wordcloud image generation. Without it, the report records a skipped wordcloud entry and continues.

Python Version: Python 3.11 or higher

# Testing:

Run the test suite with:

```python -m unittest discover -s tests```

# TODO

------------

- [x] prevent opened graph png from disappearing
- [x] add error handling to user date inputs
- [x] make graph production per term as well as aggregated to remove scaling issues
- [x] insert all graphs into PDF report (separate from TXT file report)
- [x] use the txt report to populate the PDF report with contextual data.
- [x] add asyncio options to optimise performance - particularly for media download.
- [x] better graphing, eg percent usage over time to adjust for new channels or surges in activity (ongoing improvements and new graphs - never complete this, just add more)
- [x] make sure the above is included in the report pdf
- [x] time range selection
- [x] custom channel list
- [x] wordcloud generation of all matching messages to extract additional context, terms, and insights
- [x] set ability to group terms into single line on graph (e.g translations/transliterations)
- [x] possible feature: set default location for all downloaded media with a list file of media previously downloaded to prevent duplicate

tg-keyword-trends

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README