WeakSignalFinder

KO | EN

gitlite — search

#ai #golang #android #deep-learning #docker #java #awesome #kubernetes #awesome-list #vue #pytorch #ios #openai #open-source

WeakSignalFinder

(★ 14)

A transparent lexical pipeline for spotting emerging themes in RSS news

WeakSignalFinder Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🔍 Weak Signal Finder

Weak Signal Finder is a Python pipeline that **detects emerging themes and weak signals** from RSS news feeds. It aggregates articles by language and country, cleans and lemmatizes the text using NLP, computes word frequency scores, builds contextual semantic neighborhoods, persists every run in a local SQLite database, and exposes the final results as a dated JSON file.

It also maintains a **persistent neighborhood dictionary** in a dedicated SQLite database, enriched automatically across runs, letting you observe how each word's semantic neighborhood evolves over time. A **cross-run intensity engine** computes absolute and relative word frequencies with optional weighted aggregation. An optional **SMTP email notification** can be sent at the end of every run.

> ⭐ **Enjoying Weak Signal Finder?** Give the repo a star, [open an issue](https://github.com/LittleViewer/WeakSignalFinder/issues) if something's off, or submit a pull request, every bit of feedback helps the project move forward. See the [🤝 Contributing](#-contributing) section below to get started.

---

## 📐 Architecture

```
                rssFeed.json
                     │
                     ▼
              feed_class                ← RSS aggregation (feedparser, async)
                     │
                     ▼
              prepare_data_class        ← Cleaning, lemmatization, stopword removal (spaCy)
                     │
                     ▼
   frequency_one_word_class             ← Word intensity / frequency scoring
   contextual_neighborhood_class        ← Semantic neighborhood computation
                     │
                     ▼
              api_local_class           ← Output as a dated local JSON file
                     │
                     ▼
   read_data_class                      ← Reads local_api/ snapshots,
                                          extracts neighborhoods + intensity data
                     │
                     ├──► enter_data_dictionnary_class
                     │         ← Inserts new words and neighbor triplets
                     │           into the persistent dictionary
                     │
                     └──► intensity_db_word_engine_class
                              ← Cross-run word intensity computation:
                                per-corpus relative frequency, then
                                weighted multi-corpus aggregation
                     │
                     ▼
              email_smtp_class          ← Optional end-of-run SMTP notification
```

```
   Support modules:
     • config_tool_class        → TOML configuration loader
     • log_class                → Logging, job ID generation, legacy outputs
     • utils_class              → File I/O, path normalization, type checks
     • date_utils_tool_class    → Date comparisons, cooldown enforcement
     • prepare_request_class    → SQLite helper (connect, insert, delete, query)
     • routerClass              → Dynamic class router: scans the project tree, auto-discovers
                                   every class (AST-based), and blocks any file that calls a
                                   denylisted function (eval, exec, os.system, pickle.load, ...)
                                   before it can be wired in
     • verbose_initialize       → Startup diagnostics: checks both databases and every input
                                   file exist, warns on oversized files, checks connectivity,
                                   prints the welcome banner (skippable with --silent_mode)
     • install_class            → Powers `main.py --install`, recreates missing folders and
                                   both SQLite databases from their schema files automatically
     • exit_programs            → Registers a graceful shutdown message on program exit,
                                   and wipes the runtime cache folder on the way out
     • cache_gestion            → Per-run cache buffer: every module reports a one-line stat
                                   during the run, later stitched together into the
                                   end-of-run recap email
```

```
   Two SQLite databases, tagged with a per-run job_id:

• db_Weak_Signal_Finder.db
         Tables: jobIdDateTime, saveData, idCountryFeed, feedLinkRss

• dictionnaryWord.db
         Tables: dictionnary, word, run, know_folder,
                 last_seen, intensity_word,
                 run_integrated_intensity_word,
                 multiple_intensity_word,
                 run_global_calcul_intensity
```

![UML plan of WeakSignalFinder](https://raw.githubusercontent.com/LittleViewer/WeakSignalFinder/main/docs/ulmvone.png)

---

## 🖥️ System Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| Python | 3.9 | 3.11+ |
| RAM | 2 GB | 4 GB+ |
| OS | Windows / Linux / macOS | Linux / macOS |

> spaCy `_md` language models can be memory-intensive, especially when processing large volumes of articles simultaneously. 4 GB+ of RAM is recommended if you monitor more than 10 feeds.

---

## ⚙️ Installation

### 1. Clone the repository

```bash
git clone https://github.com/LittleViewer/WeakSignalFinder.git
cd WeakSignalFinder
```

### 2. Install the dependencies

`requirements.txt` pulls spaCy **and** the medium models for English, French, German, Spanish, and Russian:

```bash
pip install -r requirements.txt
```

> **Note on package names.** The `requirements.txt` ships `dotenv`; depending on your environment you may need to replace it with `python-dotenv`. Both refer to the same library but some package managers only recognize the latter.

To support an additional language beyond the preinstalled set, download the corresponding spaCy model. The mapping is defined in `libCore/input/languageModel.json`. For example:

```bash
python -m spacy download it_core_news_md
```

Then add the entry to `languageModel.json`:

```json
{ "language": "Italian", "code_language": "it", "model_name": "it_core_news_md" }
```

### 3. Create the two SQLite databases

> ⚠️ **The repository does not ship with prebuilt `.db` files.** You must create both databases and apply the schemas before the first run. This step is **mandatory**, the pipeline will not start without them.

> 💡 **Automatic setup.** Instead of the manual steps below, you can simply run:
> ```bash
> python main.py --install
> ```
> This recreates every missing folder and both `.db` files straight from their schema files (`initial_schema` in the TOML), then asks you to restart the program. It's the fastest path for a first install or for repairing a broken environment. The manual method below is still fully supported and useful if you want more control over the process.

The project uses **two** independent SQLite databases:

| Database file | Role | Schema |
|---|---|---|
| `database_rss_run/database_don_t_touch/db_Weak_Signal_Finder.db` | Per-run pipeline state (job IDs, raw snapshots, intensity scores, neighborhoods, feed metadata) | `database_rss_run/request/schema_db.sql` |
| `dictionnary_neighbord/database/dictionnaryWord.db` | Persistent cross-run dictionary, intensity aggregation, and tracking tables | `dictionnary_neighbord/database/schema_table.sql` |

Create them both and load the schemas from the project root.

**Linux / macOS:**

```bash
# Main run database
sqlite3 database_rss_run/database_don_t_touch/db_Weak_Signal_Finder.db \
  < database_rss_run/request/schema_db.sql

# Dictionary database
sqlite3 dictionnary_neighbord/database/dictionnaryWord.db \
  < dictionnary_neighbord/database/schema_table.sql
```

**Windows (PowerShell):**

```powershell
Get-Content database_rss_run\request\schema_db.sql |
  sqlite3 database_rss_run\database_don_t_touch\db_Weak_Signal_Finder.db

Get-Content dictionnary_neighbord\database\schema_table.sql |
  sqlite3 dictionnary_neighbord\database\dictionnaryWord.db
```

**Pure Python (works everywhere, no `sqlite3` CLI required):**

```bash
python -c "
import sqlite3
sqlite3.connect('database_rss_run/database_don_t_touch/db_Weak_Signal_Finder.db') \
  .executescript(open('database_rss_run/request/schema_db.sql').read())
"

python -c "
import sqlite3
sqlite3.connect('dictionnary_neighbord/database/dictionnaryWord.db') \
  .executescript(open('dictionnary_neighbord/database/schema_table.sql').read())
"
```

After this step you should have:

```
database_rss_run/database_don_t_touch/db_Weak_Signal_Finder.db
dictionnary_neighbord/database/dictionnaryWord.db
```

The dictionary schema seeds initial rows in the `run`, `run_global_calcul_intensity`, and `run_integrated_intensity_word` tables (all with `jobId="0"` and a `1970-01-01` date) so that the very first execution considers every cooldown elapsed and triggers a full update. This is intentional.

> 💡 To wipe all data and start fresh without recreating the files, run `erase_all_table.sql` (run DB) and `erase_table.sql` (dictionary DB) the same way.

### 4. (Optional) Set up email notifications

See the [📧 Email Notifications (SMTP)](#-email-notifications-smtp) section below. Set `authorize_run = false` in the TOML and skip this step if you don't need it.

---

## 🚀 Usage

Once the databases are in place, launch the engine from the project root:

```bash
python main.py --engine_run
```

> **Run from the project root.** Paths declared in `config_weakSignalFinder.toml` are resolved relative to the current working directory. Running from another directory will not find the input files or the databases.

### Available launch modes

| Flag | Status | Description |
|---|---|---|
| `--engine_run` | ✅ Stable | Runs the full NLP pipeline (feed aggregation → analysis → dictionary → email). |
| `--endpoint_user` | 🚧 Work in progress | Interactive terminal interface for querying the dictionary and computing word concentration scores. Not yet feature-complete. |
| `--install` | ✅ Stable | Automatically (re)creates missing folders and both SQLite databases from their schema files. Exits after completion, no session is logged for this run. |
| `--silent_mode` | ✅ Stable | Skips the startup diagnostics banner (database/file/internet checks and welcome message) for a quieter, script-friendly launch. Combine with any other flag, e.g. `python main.py --engine_run --silent_mode`. |

If no flag is provided, the program prints the available options and exits.

### Startup diagnostics

Every launch (except `--install`) now runs a quick pre-flight check before the pipeline starts:

1. **Database check**, confirms both `.db` files exist; exits with a clear message if either is missing (pointing you to `--install`).
2. **Input file check**, confirms `rssFeed.json`, `languageModel.json`, `stopword.txt`, and `exclude_file.txt` are all present.
3. **Oversized file warning**, flags any of the above files heavier than `warning_size_object` GB (default: `2`) in the TOML.
4. **Internet check**, verifies connectivity with a short socket probe; exits if unreachable, since RSS aggregation needs it.
5. **Welcome banner**, prints the current date/time as a friendly session start marker.

Pass `--silent_mode` to skip the printed feedback for steps 1–3 and 5 (the checks themselves still run). A graceful "Goodbye" message is also now printed automatically whenever the program exits, success or failure.

### What happens during a run (`--engine_run`)

1. A unique `job_id` is generated and registered in the run database.
2. The RSS feed list is read from `libCore/input/rssFeed.json`.
3. All articles are fetched and parsed asynchronously.
4. Text is cleaned and lemmatized (spaCy POS filter: nouns, proper nouns, verbs, adjectives only).
5. Word frequency scores and semantic neighborhoods are computed.
6. Every intermediate state is persisted in the run database (tagged with the `job_id`).
7. The final result is written to `local_api/YYYY_M_D.local_api.txt`.
8. **Dictionary cooldown check.** If enough days have elapsed since the last dictionary update (see `cooldown_day_launch_dictionnary`), the pipeline reads the `local_api/` outputs and enriches the persistent neighborhood dictionary.
9. **Intensity engine.** If `authorize_run_intensity_engine = true`, per-word absolute and relative frequencies are stored in `intensity_word`. When the intensity cooldown (`cooldown_day_launch_inter_calcul_intensit_word`) is also met, a weighted multi-corpus aggregation is computed and stored in `multiple_intensity_word`.
10. **Notification email** is sent if `authorize_run = true` in `[parameter .email_auto]`.

### Output format

The output is a newline-delimited JSON file. Each line is a full dated snapshot:

```json
{
  "time": "[2024-01-15 08:30:00]",
  "time_jobid": "[202411300123456]",
  "intensity_word": {
    "climate": 42,
    "energy": 31,
    "transition": 18
  },
  "contextual_neighborhood": {
    "before":   [["energy", "climate"], ["new", "policy"]],
    "beetween": [["green", "energy", "policy"], ["rapid", "climate", "shift"]],
    "after":    [["climate", "change"], ["policy", "reform"]]
  },
  "word_central_neighborhood": {
    "climate": { "before": ["energy", "new"], "after": ["change", "policy"] },
    "energy":  { "before": ["green", "cheap"], "after": ["transition", "crisis"] }
  }
}
```

| Field | Type | Description |
|---|---|---|
| `time` | string | Human-readable timestamp of the run. |
| `time_jobid` | string | Unique job identifier for the run. |
| `intensity_word` | object | Words appearing more than once, with their frequency score. |
| `contextual_neighborhood` | object | All word pairs/triplets found in `before`, `beetween`, and `after` positions. |
| `word_central_neighborhood` | object | For each word: its unique direct neighbors (left and right). |

---

## 🧑‍💻 Interactive Terminal, `--endpoint_user` (Work in Progress)

> 🚧 **This mode is functional but still under active development.** The menu structure and available calculations may change between versions.

The `--endpoint_user` flag launches an interactive terminal interface that lets you **query the dictionary database directly** and run analytical calculations on accumulated data, without triggering a new pipeline run.

```bash
python main.py --endpoint_user
```

### Main menu

Once launched, you are presented with a navigation menu offering four options:

| Option | Description |
|---|---|
| **calcul** | Run analytical calculations on the dictionary data (see below). |
| **request** | Query the dictionary directly: get quick stats (unique words, unique neighbors, average neighbors per word) and search for words matching a pattern (`find_by_pattern`). |
| **help** | Browse built-in documentation explaining each calculation and its output. |
| **about** | Display session metadata: current date, time, job ID, license, and repository link. |
| **exit** | Close the program. |

After any action, a sub-menu lets you return to the main menu or exit.

#### `request` → quick stats and pattern search

The **request** option surfaces two things without leaving the terminal:

- An instant summary: total unique words in the dictionary, total unique neighbor triplets, and the average number of neighbors per word.
- **find_by_pattern**: type a prefix (e.g. `w`, `wa`, `war`) to list every distinct word in the dictionary starting with it. You can keep searching new patterns in a loop until you choose to return to the main menu.

### Word concentration score

The only calculation currently available is **concentration**, accessible from the `calcul` menu. It computes a **saliency score** for a given word by combining its intensity at a specific point in time with its presence across all historical corpora.

**Step-by-step workflow:**

1. You select a **date** from the list of past multi-corpus intensity runs (`run_global_calcul_intensity`).
2. You select a **job ID** associated with that date.
3. You pick a **word** from the `multiple_intensity_word` table for that run (or type `w` to list all available words).
4. The tool retrieves every `relative_value` entry for that word across all corpora stored in `intensity_word`.
5. Each per-corpus relative value is divided by the word's aggregated intensity from the selected run.
6. The final score is: `average(all_ratios) / number_of_corpora`.

**Interpreting the score:**

| Score range | Signal type | Example |
|---|---|---|
| Very low (e.g. `0.002`) | **Recurring signal**, the word appears consistently and broadly across many corpora. | *war*, Average: 1.91 on 755 inputs → score ≈ 0.0025 |
| Medium (e.g. `0.74`) | **Rare signal**, the word appears in few corpora but with notable intensity. | *moon*, Average: 32.69 on 44 inputs → score ≈ 0.74 |
| High (e.g. `89.9`) | **Weak signal**, the word is concentrated in very few corpora with high relative weight, warranting further investigation. | *write*, Average: 359.63 on 4 inputs → score ≈ 89.91 |

A high saliency score does not mean the word is important in absolute terms, it means the word's distribution across corpora is highly uneven, which is precisely the signature of an emerging or niche topic that hasn't yet reached mainstream coverage.

### Prerequisites

The `--endpoint_user` mode reads from the **dictionary database** (`dictionnaryWord.db`). It requires that at least one full `--engine_run` has completed with the intensity engine enabled, and that at least one multi-corpus aggregation has been computed (controlled by `cooldown_day_launch_inter_calcul_intensit_word`). If the `multiple_intensity_word` or `intensity_word` tables are empty, the concentration calculation will have no data to work with.

### Planned improvements

This mode is intended to grow into a full local query interface for the dictionary. Future additions may include neighborhood exploration, word trajectory visualization, and custom filtering by date range or corpus.

---

## 📚 Persistent Neighborhood Dictionary

The `dictionnary_neighbord/` module is the **long-term memory** of the tool. While each run produces a snapshot of the current semantic landscape in `local_api/`, the dictionary database accumulates these snapshots over time so you can study how the neighborhood of any given word evolves.

### How it works

At the end of every run, `main.py` checks the `run` table of `dictionnaryWord.db` via `date_utils_tool_class`. If the difference (in days) between today and the most recent entry is **greater than or equal to** `cooldown_day_launch_dictionnary`, a dictionary update is triggered. Otherwise it is skipped.

When an update is triggered:

1. `read_data_class` scans `local_api/` for every dated output file, **except** those listed in `dictionnary_neighbord/exclude_file.txt` and those already recorded in the `know_folder` table.
2. It picks the JSON field configured by `part_of_local_api` (default: `word_central_neighborhood`) from every snapshot and merges all blocks into a single deduplicated structure (one `before` set and one `after` set per central word). It simultaneously extracts `intensity_word` data from each snapshot, keyed by `job_id`.
3. `enter_data_dictionnary_class` opens `dictionnaryWord.db`:
   - New central words are inserted into the `word` table, tagged with the current `job_id`.
   - New `(central_word, position, neighbor)` triplets are inserted into the `dictionnary` table. Existing triplets are skipped.
   - Words and triplets that already exist are recorded in the `last_seen` table with a timestamp and type, so you can track when known items were last encountered.
   - Consumed file names are recorded in `know_folder` to avoid re-reading.
   - The current `job_id` and date are recorded in the `run` table for cooldown enforcement.

### Database schema (`dictionnaryWord.db`)

The dictionary database contains **nine** tables:

```sql
CREATE TABLE IF NOT EXISTS dictionnary(
   central_word TEXT,
   position_     TEXT,        -- "before" or "after"
   word_neighbor TEXT,
   run_added     TEXT         -- job_id that inserted this triplet
);

CREATE TABLE IF NOT EXISTS word(
   word      TEXT,
   run_added TEXT,
   PRIMARY KEY(word),
   FOREIGN KEY(word) REFERENCES dictionnary(central_word)
);

CREATE TABLE IF NOT EXISTS run(
   jobId TEXT,
   date_ TEXT,
   PRIMARY KEY(jobId)
);

CREATE TABLE IF NOT EXISTS know_folder(
   jobId       TEXT,
   name_folder TEXT,
   PRIMARY KEY(name_folder)
);

CREATE TABLE IF NOT EXISTS last_seen(
   date      TEXT,
   jobId     TEXT,
   type_seen TEXT,     -- "central_word" or "neighbord_word"
   data      TEXT,
   FOREIGN KEY (jobId) REFERENCES run(jobId)
);

CREATE TABLE IF NOT EXISTS intensity_word(
   jobId          TEXT,
   word           TEXT,
   absolute_value INTEGER,
   relative_value REAL,
   FOREIGN KEY (jobId) REFERENCES run_integrated_intensity_word(jobId),
   FOREIGN KEY (word)  REFERENCES word(word)
);

CREATE TABLE IF NOT EXISTS run_integrated_intensity_word(
   jobId TEXT,
   date  TEXT,
   PRIMARY KEY(jobId)
);

CREATE TABLE IF NOT EXISTS multiple_intensity_word(
   jobId                TEXT,
   word                 TEXT,
   absolute_value       INTEGER,
   relative_value       REAL,
   calcul               TEXT,
   FOREIGN KEY (jobId) REFERENCES run_integrated_intensity_word(jobId),
   FOREIGN KEY (word)  REFERENCES word(word)
);

CREATE TABLE IF NOT EXISTS run_global_calcul_intensity(
   jobId TEXT,
   date_ TEXT,
   PRIMARY KEY(jobId)
);
```

### Tuning the cooldowns

Cooldowns are set under `[parameter .for_launch]` in the TOML:

```toml
[parameter .for_launch]
cooldown_day_launch_dictionnary = 1
cooldown_day_launch_inter_calcul_intensit_word = 7
```

| Parameter | Default | Effect |
|---|---|---|
| `cooldown_day_launch_dictionnary` | `1` | Minimum days between two dictionary updates. `0` = every run. |
| `cooldown_day_launch_inter_calcul_intensit_word` | `7` | Minimum days between two cross-corpus intensity aggregations. |

### Choosing the source field

```toml
[parameter .for_launch]
part_of_local_api = "word_central_neighborhood"
```

The dictionary loader expects a `{ word: { before: [...], after: [...] } }` shape. Changing this value without adapting the loader will break the dictionary update.

### Excluding output files

`dictionnary_neighbord/exclude_file.txt` lists, one per line, the file names from `local_api/` that the dictionary loader must ignore. Defaults: `test`, `placeholder`.

---

## 📊 Cross-Run Intensity Engine

The intensity engine (`intensity_db_word_engine_class`) adds a **quantitative dimension** to the dictionary's qualitative neighborhood data. It is enabled by default and runs as part of the dictionary update step.

### How it works

1. **Per-corpus storage.** For every `local_api/` snapshot consumed, the engine extracts each word's absolute count and computes its relative frequency within that corpus (`absolute / total_words`). These values are stored in `intensity_word`, one row per `(jobId, word)`.

2. **Multi-corpus aggregation.** When the intensity cooldown (`cooldown_day_launch_inter_calcul_intensit_word`) is met, the engine reads all rows from `intensity_word`, groups them by word, and computes:
   - `total_occurence`, sum of absolute values across all corpora.
   - `total_relative_value`, sum of relative frequencies.
   - `number_corpus`, number of corpora in which the word appeared.
   - `total_relative_value_weighted`, `total_relative_value / number_corpus`.

The result is stored in `multiple_intensity_word`.

### Configuration

```toml
[parameter .for_launch]
authorize_run_intensity_engine = true
single_multiple_intensity_save = false

[parameter .optimize]
batch_longer_list = 5000
```

| Parameter | Default | Effect |
|---|---|---|
| `authorize_run_intensity_engine` | `true` | Set to `false` to disable the intensity engine entirely. |
| `single_multiple_intensity_save` | `false` | When `true`, the `multiple_intensity_word` table is wiped before each new aggregation (useful on low-memory systems). When `false`, rows accumulate. |
| `batch_longer_list` | `5000` | Number of INSERT rows per batch commit. Higher values are faster but use more memory. |

---

## 🗄️ Run Cache (Email Digest Buffer)

`libCore/cache_gestion.py` ✨ New is the small mechanism that collects, over the course of a single run, the one-line stats you see in the final notification email under "Stat of Run". It is not a performance cache (nothing is read back to skip work), it's a scratch buffer scoped to a single `job_id`.

### What it means for you (user)

You don't need to configure or touch this yourself, it's fully automatic and only kicks in if you actually want the recap email:

- If `[parameter .email_auto] authorize_run = true`, a small file is created at the very start of the run in `libCore/cache_service_wsf/`, named after the current `job_id`.
- As the pipeline progresses, each stage (word frequency, contextual neighborhood, dictionary enrichment, intensity aggregation) drops a short human-readable line into that file, e.g. *"Intensity Word: 128 intensity words counter."*
- At the end of the run, all the collected lines are read back, formatted as a bullet list, and inserted into the success email you receive (see [📧 Email Notifications](#-email-notifications-smtp) below).
- The file itself never lingers, `exit_programs.py` deletes the whole `cache_service_wsf/` folder automatically whenever the program exits, success or failure. Nothing accumulates between runs, and there is nothing for you to clean up manually.
- If `authorize_run = false`, most of these calls are skipped entirely (via `wrapper_add_data_in_file`), so disabling email also avoids the small amount of extra disk I/O.

### Why it matters for contributors (dev)

The current use case is intentionally narrow, one buffer, one consumer (the end-of-run email), populated by a handful of hardcoded call sites (`frequency_one_word_class`, `contextual_neighborhood_class`, `enter_data_dictionnary_class`, `intensity_db_word_engine_class`). That narrowness is where the potential lies for anyone looking to contribute:

- **Generic per-run scratch space.** The API (`pipe_create_cache_file`, `pipe_add_data_content_file`, `get_content_cache_file`, `wrapper_add_data_in_file`) is already file-name and `job_id`-agnostic, it could back other per-run artifacts beyond the email digest (e.g. a run summary written to disk, a webhook payload, a `--endpoint_user` "last run" view) without touching the core pipeline.
- **Structured content instead of plain lines.** Cache entries are currently free-form strings joined with `\n`. Moving to a structured format (JSON lines, for instance) would make the recap machine-readable, useful for a future dashboard, a Slack/Discord notifier, or richer HTML emails.
- **More granular gating.** Today the whole mechanism is tied to `email_auto.authorize_run`. A dedicated flag would let the cache (and whatever consumes it) run independently of whether an email is actually sent.
- **Error-path coverage.** The success email builds its digest from the cache, but the separate error email (triggered from `log_class` on `ERROR`/`CRITICAL`) currently doesn't pull from it, there's room to enrich error notifications with whatever stats had already been collected before the failure.

If any of this is interesting, it's a self-contained module with a small surface area, a good first contribution. See [🤝 Contributing](#-contributing) to get started.

---

## 📧 Email Notifications (SMTP)

The pipeline can send a **notification email at the end of every run**. It is optional and built on the Python standard library (`smtplib`) plus `python-dotenv` for the password.

### What it sends

Two kinds of emails can be sent, both built from the same `email_smtp_class`:

- **Success notification**, sent once at the end of a completed `--engine_run`. It contains the run's timestamp, the `job_id`, a "Stat of Run" digest, a link to the project repository, and a copyright line. The digest is a bullet list assembled from the [run cache](#-run-cache-email-digest-buffer) and typically includes lines such as:
  - the number of intensity words counted,
  - the contextual neighborhood stats (words found before/between/after a central word),
  - how many new entries were added to the persistent dictionary (only when a dictionary update actually ran),
  - how many multi-corpus intensity rows were saved (only when the intensity cooldown was also met).
- **Error notification**, sent independently, any time an `ERROR` or `CRITICAL` event is logged during the run (from `log_class`), with the severity, the function that raised it, and the message. This can fire zero, one, or several times per run, and is not tied to the run cache.

If the email cannot be prepared or sent, the pipeline does **not** fail, it logs an `ERROR` and finishes normally.

### Configuration

```toml
[parameter .email_auto]
authorize_run = true
server        = "smtp.gmail.com"
port          = 587
sender        = "[EMAIL_ADDRESS]"
password_file = "password_app.env"
receiver      = "[EMAIL_ADDRESS]"
```

| Key | Role |
|---|---|
| `authorize_run` | `true` enables the email, `false` disables it entirely. |
| `server` | SMTP server hostname. Gmail = `smtp.gmail.com`. |
| `port` | SMTP submission port. Use `587` (STARTTLS). The implementation does **not** support implicit TLS on `465`. |
| `sender` | The "from" address, also used for authentication. |
| `password_file` | Path (relative to project root) of the `.env` file containing `PASSWORD_EMAIL`. |
| `receiver` | The "to" address. Can be the same as `sender`. |

### Setup (Gmail example)

1. **Enable 2-Step Verification** on your Google account.
2. **Generate a Google App Password** at <https://myaccount.google.com/apppasswords>, pick "Mail". You'll get a 16-character code.
3. **Create the password file** from the shipped template:
   ```bash
   cp archive_configuration/password_app.env.archive password_app.env
   ```
4. **Edit `password_app.env`** and replace the placeholder:
   ```env
   PASSWORD_EMAIL=abcd efgh ijkl mnop
   ```
5. **Fill `sender` and `receiver`** in `config_weakSignalFinder.toml`.

For other providers, adjust `server`, `port`, `sender`, and `PASSWORD_EMAIL`. The server must accept STARTTLS on the chosen port.

### 🔒 Security notes

- **Never commit `password_app.env`.** The `.gitignore` already excludes `*.env` and `.env`.
- **Use an App Password, not your account password.**
- **The TOML is committed, the password is not.** If `sender`/`receiver` are personal, keep the repo private or replace them with placeholders before publishing.
- **Revoke and rotate.** If the password file leaks, revoke the App Password from your provider and generate a new one.
- **Disable when unused.** Set `authorize_run = false` to fully skip the SMTP step.

---

## 🛠️ Configuration

All runtime configuration lives in a single TOML file at the project root, alongside the input files used by the NLP pipeline.

### Main configuration, `config_weakSignalFinder.toml`

Loaded once at startup by `libCore/config_tool_class.py` and consumed by every module via `key_return(table, key, sub_table)`.

```toml
[path]

[path .class_feed]
extract_feed_Path = "libCore\\input\\rssFeed.json"

[path .api_local]
open_file = "local_api\\"

[path .log]
save_data_set = "dataset\\"
save_state    = "saveState\\"
log_file      = "log\\"

[path .prepare_data]
file_model    = "libCore\\input\\languageModel.json"
file_stopword = "libCore\\input\\stopword.txt"

[path .read_data_dictionnary]
exclude_file = "dictionnary_neighbord\\exclude_file.txt"
dataset      = "local_api\\"

[path .database]
database_run_sqlite         = "database_rss_run\\database_don_t_touch\\db_Weak_Signal_Finder.db"
database_dictionnary_sqlite = "dictionnary_neighbord\\database\\dictionnaryWord.db"

[filename]

[filename .cache]
email_content = "email_final_sender"

[parameter]

[parameter .global_program]
start_file = "WeakSignalFinder"
argument_run = ["--silent_mode", "--endpoint_user", "--install", "--engine_run"]
warning_size_object = 2

[parameter .install]
initial_schema = {db_Weak_Signal_Finder = "database_rss_run\\request\\schema_db.sql", dictionnaryWord = "dictionnary_neighbord\\database\\schema_table.sql"}

[parameter .endpoint_user]
initial_menu = {calcul = "calcul_parameter", request = "request_parameter", help = "help_parameter", about = "about_parameter", exit = "exit_parameter"}
list_calcul = {concentration = "concentration_word_calcul"}
list_request = {find_by_pattern = "find_word_pattern"}

[parameter .frequency_one_word]
filter_word = 1

[parameter .for_launch]
cooldown_day_launch_dictionnary              = 1
cooldown_day_launch_inter_calcul_intensit_word = 7
part_of_local_api                            = "word_central_neighborhood"
authorize_run_intensity_engine               = true
single_multiple_intensity_save               = false

[parameter .email_auto]
authorize_run = true
server        = "smtp.gmail.com"
port          = 587
sender        = "[EMAIL_ADDRESS]"
password_file = "password_app.env"
receiver      = "[EMAIL_ADDRESS]"

[parameter .optimize]
batch_longer_list = 5000
```

**Full parameter reference:**

| Section | Key | Role |
|---|---|---|
| `[path .class_feed]` | `extract_feed_Path` | Path to the JSON list of RSS feeds. |
| `[path .api_local]` | `open_file` | Folder where dated `*.local_api.txt` outputs are written. |
| `[path .log]` | `log_file` / `save_state` / `save_data_set` | Folders for logs, legacy savestate, and legacy dataset files. |
| `[path .prepare_data]` | `file_model` / `file_stopword` | spaCy language-model mapping and stopword list. |
| `[path .read_data_dictionnary]` | `exclude_file` / `dataset` | Dictionary exclusion list and the folder it reads from. |
| `[path .database]` | `database_run_sqlite` | Path to the per-run pipeline SQLite database. |
| `[path .database]` | `database_dictionnary_sqlite` | Path to the persistent dictionary SQLite database. |
| `[parameter .global_program]` | `start_file` | Root package name used by the dynamic class router to resolve module imports. |
| `[parameter .global_program]` | `argument_run` | List of accepted CLI flags (`--silent_mode`, `--endpoint_user`, `--install`, `--engine_run`). |
| `[parameter .global_program]` | `warning_size_object` | Size threshold, in GB, above which the startup check warns about an oversized input/database file. Default: `2`. |
| `[parameter .install]` | `initial_schema` | Maps each database name to the `.sql` schema file used by `--install` to (re)build it. |
| `[parameter .endpoint_user]` | `initial_menu` / `list_calcul` / `list_request` | Menu labels and available calculations/requests for the `--endpoint_user` terminal. |
| `[filename .cache]` ✨ New | `email_content` | Base name used for the per-run cache file that buffers stat lines for the end-of-run email (see [🗄️ Run Cache](#-run-cache-email-digest-buffer)). Default: `email_final_sender`. |
| `[parameter .frequency_one_word]` | `filter_word` | Minimum frequency floor. Words with count ≤ this value are dropped. Default: `1`. |
| `[parameter .for_launch]` | `cooldown_day_launch_dictionnary` | Days between dictionary updates. Default: `1`. |
| `[parameter .for_launch]` | `cooldown_day_launch_inter_calcul_intensit_word` | Days between cross-corpus intensity aggregations. Default: `7`. |
| `[parameter .for_launch]` | `part_of_local_api` | JSON field consumed by the dictionary loader. Default: `word_central_neighborhood`. |
| `[parameter .for_launch]` | `authorize_run_intensity_engine` | Enable (`true`) or disable (`false`) the cross-run intensity engine. |
| `[parameter .for_launch]` | `single_multiple_intensity_save` | `true` = wipe `multiple_intensity_word` before each aggregation. `false` = accumulate. |
| `[parameter .email_auto]` | `authorize_run` | Enable/disable the SMTP notification. |
| `[parameter .email_auto]` | `server` / `port` / `sender` / `receiver` / `password_file` | SMTP connection details (see Email section). |
| `[parameter .optimize]` | `batch_longer_list` | Rows per batch INSERT commit for the intensity engine. Default: `5000`. |

> Paths use Windows-style backslashes (`\\`) but are normalized internally, the file works on Windows, Linux, and macOS.

A pristine copy is shipped as `archive_configuration/config_weakSignalFinder.toml.archive`. A `.env` template is at `archive_configuration/password_app.env.archive`.

### RSS Feeds, `libCore/input/rssFeed.json`

Each entry has a source name, a feed URL, and a country/language code:

```json
[
  {
    "name_organization": "Le Monde",
    "rss_link": "https://www.lemonde.fr/rss/une.xml",
    "country": "fr"
  },
  {
    "name_organization": "BBC News",
    "rss_link": "https://feeds.bbci.co.uk/news/rss.xml",
    "country": "en"
  }
]
```

### Language Models, `libCore/input/languageModel.json`

Maps language codes to spaCy model names. The shipped file supports four languages:

```json
[
  { "language": "English",  "code_language": "en", "model_name": "en_core_web_md" },
  { "language": "Spanish",  "code_language": "es", "model_name": "es_core_news_md" },
  { "language": "German",   "code_language": "de", "model_name": "de_core_news_md" },
  { "language": "French",   "code_language": "fr", "model_name": "fr_core_news_md" }
]
```

> **Note.** `requirements.txt` also installs the Russian model (`ru_core_news_md`). To use it, add a corresponding entry here with `"code_language": "ru"`.

### Stopwords, `libCore/input/stopword.txt`

One word per line. Pre-populated with English/French structural words plus a thematic block tied to politics, media, and institutions. Edit freely to fit your monitoring scope, anything listed here is excluded from analysis on top of the spaCy POS filter.

### Dictionary exclusions, `dictionnary_neighbord/exclude_file.txt`

One file name per line. Any file in `local_api/` whose name appears here is skipped during dictionary enrichment. Defaults: `test`, `placeholder`.

---

## 📋 Run Outputs

Every run writes to several places, all tagged with the same `job_id`:

| Storage | Location | Role |
|---|---|---|
| **Run SQLite DB** | `database_rss_run/database_don_t_touch/db_Weak_Signal_Finder.db` | Primary per-run store. Tables: `jobIdDateTime`, `saveData`, `idCountryFeed`, `feedLinkRss`. |
| **Dictionary SQLite DB** | `dictionnary_neighbord/database/dictionnaryWord.db` | Cross-run memory. Tables: `dictionnary`, `word`, `run`, `know_folder`, `last_seen`, `intensity_word`, `run_integrated_intensity_word`, `multiple_intensity_word`, `run_global_calcul_intensity`. |
| `YYYY_M_D.local_api.txt` | `local_api/` | Final consumable JSON output; also consumed as input by the dictionary loader. |
| `YYYY_M_D.log.txt` | `log/` | Execution trace with severity levels (`INFO`, `WARN`, `ERROR`, `CRITICAL`). |
| `YYYY_M_D.savestate.txt` | `saveState/` | *Legacy file output*, kept for backward compatibility (see `LEGACY_FUNCTION.md`). |
| `YYYY_M_D.dataset.txt` | `dataset/` | *Legacy file output*, same as above. |
| End-of-run email | `receiver` mailbox | *Optional.* Sent only if `authorize_run = true`. |

> ⚠️ Do **not** open either `.db` file with another tool while the pipeline runs, risk of corruption.

Each log entry is a JSON object:

```json
{
  "content": "The word intensity is calculated as well as saved in the dataset.",
  "severity": "INFO",
  "function_call": "frequency_one_word() : pipe_frequency_one_word()",
  "timestamp": "2024-01-15 08:30:12.456789"
}
```

---

## ⚠️ Known Limitations

Silent skips on bad input, a frequency floor on rare words, unbounded growth of the dictionary/intensity tables over long periods, and a hard requirement for both databases to exist (now easily fixed with `--install`). Nothing blocking, just plan your storage and cooldowns accordingly.

---

## 📁 Project Structure

```
weak-signal-finder/
├── main.py                                           ← Entry point (--engine_run, --endpoint_user, --install, --silent_mode)
├── core_engine_pipe.py                               ← Full NLP pipeline orchestration
├── endpoint_user_pipe.py                             ← Interactive terminal (--endpoint_user, WIP)
├── verbose_initialize_wsf.py                         ← Startup diagnostics (DB/file/internet checks, banner)
├── exit_programs.py                                  ← Graceful shutdown message on exit, also wipes the run cache
├── config_weakSignalFinder.toml
├── requirements.txt
├── password_app.env                                  ← Created by you, NOT committed
├── routerClassPackage/
│   └── routerClass.py                                ← Dynamic class auto-discovery & security denylist
├── install/
│   └── install_pipe_class.py                         ← Powers `--install` (auto DB/folder creation)
├── archive_configuration/
│   ├── config_weakSignalFinder.toml.archive
│   └── password_app.env.archive
├── libCore/
│   ├── feed_class.py                                 ← RSS fetching (feedparser, async)
│   ├── prepare_data_class.py                         ← Text cleaning, spaCy lemmatization
│   ├── frequency_one_word_class.py                   ← Word frequency scoring
│   ├── contextual_neighborhood_class.py              ← Semantic neighborhood computation
│   ├── api_local_class.py                            ← JSON output writer
│   ├── email_smtp_class.py                           ← SMTP notification
│   ├── cache_gestion.py                              ← ✨ New: per-run cache buffer feeding the end-of-run email
│   ├── log_class.py                                  ← Logging, job ID generation
│   ├── config_tool_class.py                          ← TOML configuration loader
│   ├── argument_start_class.py                       ← CLI argument parser
│   ├── utils_class.py                                ← File I/O, path, type utilities
│   ├── date_utils_tool_class.py                      ← Date comparison, cooldown logic
│   ├── cache_service_wsf/                             ← ✨ New: runtime-only folder, created and deleted automatically
│   └── input/
│       ├── rssFeed.json                              ← Feed definitions
│       ├── languageModel.json                        ← Language → spaCy model mapping
│       └── stopword.txt                              ← Stopword list
├── database_rss_run/
│   ├── prepare_request_class.py                      ← SQLite helper (connect, query, insert)
│   ├── database_don_t_touch/
│   │   └── db_Weak_Signal_Finder.db                  ← Created by you
│   └── request/
│       ├── schema_db.sql
│       └── erase_all_table.sql
├── dictionnary_neighbord/
│   ├── read_data_class.py                            ← Reads local_api/ snapshots
│   ├── enter_data_dictionnary_class.py               ← Inserts words & neighbors into dictionary
│   ├── intensity_db_word_engine_class.py             ← Cross-run intensity computation
│   ├── exclude_file.txt
│   └── database/
│       ├── dictionnaryWord.db                        ← Created by you
│       ├── schema_table.sql
│       └── erase_table.sql
├── endpoint_user_core/                               ← Interactive terminal modules (WIP)
│   ├── interaction_user_class.py
│   ├── calcul_class.py
│   ├── request_class.py                              ← Dictionary stats & find_by_pattern search
│   ├── utils_interaction_terminal_class.py
│   └── template/
│       ├── prepared_request.json
│       └── help_prepared.json
├── docs/
│   └── ulmvone.png                                   ← Architecture diagram
├── log/
├── saveState/
├── dataset/
└── local_api/
```

---

## 🤝 Contributing

Contributions are welcome. Before opening a pull request, please review:

- [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md), community standards.
- [`CLA.md`](CLA.md), the Individual Contributor License Agreement you implicitly accept by submitting a contribution.
- [`SECURITY.md`](SECURITY.md), how to responsibly disclose a vulnerability (do **not** open a public issue for security matters).
---

## 📄 License

This project is distributed under the [GPLv3 License](LICENSE).

---

## Disclaimer

This tool is designed as an analytical aid to support **media monitoring and weak signal analysis**. The outputs are not definitive conclusions but data points intended to guide human interpretation. The quality of the analysis depends on the RSS feeds provided as input, the tool does not verify source accuracy, it solely counts and groups words without interpreting meaning.

**A high frequency score does not imply importance.** A word appearing frequently may simply be overrepresented in the selected sources. Interpret intensity scores in light of the feeds you have configured. Lemmatization quality depends on the spaCy model used, an unsuitable model may produce incorrect lemmas.

All processing is performed locally. The tool does not collect, transmit, or store personal data. **The outputs are not anonymized.** If configured feeds contain proper nouns, names, or places, these appear as-is in the results and accumulate in the persistent dictionary over time.

**Users are solely responsible for ensuring compliance with the terms of service of each RSS feed they configure.** Some publishers prohibit automated aggregation or redistribution. This tool provides no guarantee of legal compliance.

---

## 🚀 Get Involved

If Weak Signal Finder is useful to you, consider:

- ⭐ **Starring the repository** so more people discover it.
- 🐛 **Reporting bugs or ideas** via [GitHub Issues](https://github.com/LittleViewer/WeakSignalFinder/issues).
- 🔀 **Opening a pull request**, see [`CLA.md`](CLA.md) and [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md) first.
- 🔒 **Reporting vulnerabilities privately**, per [`SECURITY.md`](SECURITY.md), never in a public issue.

Every star, issue, and PR helps keep the project moving. Thanks for checking it out!

WeakSignalFinder

WeakSignalFinder Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

WeakSignalFinder

WeakSignalFinder Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README