nse-sentiment-analyzer

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

nse-sentiment-analyzer

★ 10 Open GitHub ↗

NSE Stock Sentiment Analyzer — Enter any NSE ticker & get live price + news sentiment in one dashboard. Streamlit + yfinance + VADER. Try it: https://nse-sentiment-analyzer.streamlit.app/

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# NSE Stock Sentiment Analyzer

**Multi-source sentiment + technical indicators for NSE equities & ETFs, in one dashboard.**

<p align="center">
  <a href="https://nse-sentiment-analyzer.streamlit.app"><b>Launch App</b></a>
  &nbsp;·&nbsp;
  <a href="https://github.com/AshayK003/nse-sentiment-analyzer/issues">Report Bug</a>
  &nbsp;·&nbsp;
  <a href="mailto:darkcharon3301@gmail.com">Request Feature</a>
</p>

</div>

---

## Table of Contents

- [Overview](#-overview)
- [Architecture](#-architecture)
- [Quick Start](#-quick-start)
- [Environment Variables](#-environment-variables)
- [Project Structure](#-project-structure)
- [Local Development](#-local-development)
- [Testing](#-testing)
- [Deployment](#-deployment)
- [Troubleshooting](#-troubleshooting)
- [Contributing](#-contributing)
- [License](#-license)

---

## Overview
Enter any NSE ticker **or company name** and get a **BULLISH / NEUTRAL / BEARISH** signal backed by:

- **Smart ticker search** — type a company name ("HDFC Bank", "Tata Motors", "Zomato"), alias ("SBI", "L&T"), or partial name and get the correct NSE ticker automatically. 504 aliases, Yahoo Finance REST API (~0.5s), yfinance SDK, and direct ticker probe — chained with caching. Handles rebrands (Zomato → Eternal) and splits (Tata Motors → TMPV/TMCV).
- **Live market data** — price, change %, volume, PE ratio, and D/E ratio via Yahoo Finance
- **Interactive price chart** — 2-year candlestick chart with volume bars, 50-day SMA, 200-day SMA, and Bollinger Bands (20,2) overlays. Powered by TradingView Lightweight Charts. Zoom, pan, crosshair on hover. Visual legend identifies all overlays. Zero Python dependencies.
- **Multi-source news sentiment** — RSS feeds from Moneycontrol, Economic Times, LiveMint (Markets + Companies + Industry), NDTV Profit, Google News, with DuckDuckGo fallback
- **Event-aware scoring** — headlines classified by 19 event types (earnings, order wins, litigation, regulatory approvals, buybacks, etc.) with signed sentiment bias. Correctly scores "SEBI penalty" as negative and "SEBI clears merger" as positive — something VADER alone misses.
- **SmartScore composite (0–100)** — combines recency-weighted EWMA (36h half-life), event-adjusted sentiment, headline breadth, and news volume into a single calibrated score. See [SmartScore Algorithm](docs/SMARTSCORE.md) for implementation details, formulas, and examples.
- **Self-calibrating source weights** — each source's confidence weight learns from your 👍/👎 votes via Bayesian Beta-Binomial inference. After ~10–50 votes, weights reflect your actual accuracy experience.
- **Enhanced VADER + 123-term Indian financial lexicon** — includes Indian banking metrics (NPA, GNPA, NIM, credit growth, slippage, provisioning), profitability shorthand (PAT, EBITDA, ROE, ROCE), IPO/capital market terms (oversubscribed, undersubscribed), fund flow terms (inflow, outflow), Hinglish (tezi, mandi, tej, mand), and general financial context not in vanilla VADER. Optionally swap to FinBERT via `USE_FINBERT=true`.
- **Technical indicators** — RSI(14), SMA crossover (50/200), MACD from 2-year OHLCV history
- **Portfolio mode** — track holdings with P&L, qty/shares field, auto-fetched LTPs, and one-click clear all
- **FII/DII institutional flow** — NSE India official FII/FPI and DII data with Net value. Glassmorphism card shows latest day's flow plus a 7-day history table. Auto-saves daily snapshots.
- **VWAP + Pivot levels** — intraday fair value and classic support/resistance from yesterday's HLC
- **Cascade / Ripple Tracking** — when news mentions a macro driver (crude oil, rupee, gold, steel, natural gas, coal, sugar, aluminum), the dashboard now shows which other tickers are indirectly affected and why. Hand-curated relationship map covers 8 commodity groups and 27 tickers across OMCs, aviation, paints, IT, gold, steel, city gas, coal/power, logistics, sugar, and aluminum. **Per-ticker direction** — each ticker has its own sensitivity (e.g., ONGC benefits from crude rise while OMCs are hurt), so the display shows ticker-level Bullish/Bearish labels. **Ticker mention scanning** — only tickers actually mentioned in article text are shown. **Searched-ticker focus** — commodities not affecting the searched ticker are hidden. **Direction inferred from article text** — 40+ up/down keywords detect whether the commodity is rising or falling. Falls back to cautious Bearish when direction is unclear. **Commodity-specific RSS** — 3 dedicated feeds (Moneycontrol Commodities, ET Commodities, Google News Commodities) alongside 9 India market feeds, with DuckDuckGo fallback for light days. Appears on both the ticker analysis page and the home page. No extra network calls.
- **Adaptive Sentiment Engine (P0)** — Learns sentiment from actual price reactions, not labels. TF-IDF cluster learner groups headlines into semantic neighborhoods, tracks realized average price reaction per cluster, auto-adapts as market regimes shift (no retraining). Recalibrates weights every 72h against ground-truth price moves. Based on *Hybrid News Sentiment Engine* (arXiv:2606.03457).
- **News Dissemination Breadth Clustering (P0)** — Clusters related articles by content similarity (TF-IDF + DBSCAN) to measure dissemination breadth = market impact proxy. Large multi-source clusters = high-impact events. Dissemination score (0-1) shown in dashboard as confidence indicator. Based on *FinGPT* (arXiv:2412.10823).
- **Per-headline price reaction predictions** — Dashboard shows expected 1h/4h price moves for each headline based on adaptive cluster learning.
- **Resource protections** — per-session rate limiter (6 searches/min), auto-pruning cache (500-entry cap), DDGS fallback cooldown, and thread-safe rate-limit tracking (including DDGS) prevent quota exhaustion and race conditions under multi-user load. Configured in `.streamlit/config.toml`.
- **XSS-safe rendering** — all user data (ticker names, company names) is HTML-escaped via `html.escape()` with single-quote support. Shareable `?ticker=` URLs are validated before API calls. RSS feed URLs are checked for `http://`/`https://` scheme.
- **CSP hardened** — Content Security Policy `connect-src` restricted to known API domains (Yahoo Finance, Google News, Moneycontrol, Economic Times, LiveMint, NDTV Profit). No wildcard.
- **3-tier price history cache** — L1 memory → L2 disk (`.price_cache/` JSON files) → L3 network (yfinance). Disk cache survives Streamlit Cloud restarts with 7-day TTL, so recurring users get instant price loads instead of waiting on yfinance. Auto-evicts stale entries.

**All data sources are free and public. Zero API keys required. No registration.**

---

## Architecture

See **[ARCHITECTURE.md](ARCHITECTURE.md)** for the full system diagram, module responsibilities, data flow, and design decisions.

```
┌─────────────────────────────────────────────────────────────┐
│  app.py                   Streamlit entry point, UI layout  │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  data_fetcher.py    Yahoo Finance + RSS + DuckDuckGo  │  │
│  │                     NSE_TICKERS (271 stocks)          │  │
│  │                     Alias map (461 entries)           │  │
│  │                     Smart ticker resolution           │  │
│  │                       → local dict (instant)          │  │
│  │                       → aliases (instant)             │  │
│  │                       → Yahoo REST API (~0.5s)        │  │
│  │                       → yfinance SDK (~1s)            │  │
│  │                       → direct .NS probe (~1s)        │  │
│  └───────────────────────┬───────────────────────────────┘  │
│                          │                                    │
│  ┌───────────────────────▼───────────────────────────────┐  │
│  │  sentiment.py         VADER + 125-term Indian financial│  │
│  │                       FinBERT integration (optional)  │  │
│  │                       Source-weighted blending        │  │
│  │  ┌─────────────────────────────────────────────────┐ │  │
│  │  │  event_classifier.py    19 event types           │ │  │
│  │  │                        Signed bias per event     │ │  │
│  │  └─────────────────────────────────────────────────┘ │  │
│  │  ┌─────────────────────────────────────────────────┐ │  │
│  │  │  aggregate_sentiment.py  SmartScore 0–100        │ │  │
│  │  │                        EWMA, breadth, volume    │ │  │
│  │  └─────────────────────────────────────────────────┘ │  │
│  │  ┌─────────────────────────────────────────────────┐ │  │
│  │  │  cascade.py            8 macro drivers          │ │  │
│  │  │                        Per-ticker direction    │ │  │
│  │  └─────────────────────────────────────────────────┘ │  │
│  └───────────────────────┬───────────────────────────────┘  │
│                          │                                    │
│  ┌───────────────────────▼───────────────────────────────┐  │
│  │  indicators.py        RSI(14), SMA crossover, MACD    │  │
│  │  intraday.py          VWAP, pivot levels, India VIX   │  │
│  │  market_data.py       FII/DII flow, MMI               │  │
│  └───────────────────────┬───────────────────────────────┘  │
│                          │                                    │
│  ┌───────────────────────▼───────────────────────────────┐  │
│  │  persistence.py       JSON file I/O                   │  │
│  │                       Portfolio, track record, cache  │  │
│  │                       Source accuracy (Bayesian)      │  │
│  │                       Sentiment history (CSV)         │  │
│  └───────────────────────┬───────────────────────────────┘  │
│                          │                                    │
│  ┌───────────────────────▼───────────────────────────────┐  │
│  │  render.py            Dark HTML/CSS dashboard         │  │
│  │                       SmartScore sparkline            │  │
│  │                       Lucide SVG icons                │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

### Data Flow

1. **Input** — User types a ticker or company name (e.g., "HDFC Bank", "Zomato", "RELIANCE")
2. **Resolve** — `resolve_ticker()` maps the input to an NSE ticker via 5-tier chain: local dict → aliases → Yahoo REST → yfinance SDK → direct probe. Returns `(ticker, company_name)`.
3. **Fetch** — `get_stock_info()`, `search_news()`, and `get_fii_dii_flow()` run in parallel via `ThreadPoolExecutor(3)`. Stock data comes from a 3-tier cache: L1 (in-memory dict, instant), L2 (`.price_cache/` disk cache, survives restarts), or L3 (yfinance network call). News from RSS (9+ sources) with DuckDuckGo fallback, FII/DII from NSE India. Total fetch time: ~2s.
4. **Analyze** — `sentiment.py` scores each headline via VADER + financial lexicon, applies event-classifier corrections, then blends results using Bayesian source weights
5. **Aggregate** — `aggregate_sentiment.compute_smartscore()` produces the 0–100 SmartScore from EWMA, event-adjusted sentiment, breadth, and volume
6. **Cascade** — `cascade.detect_cascade()` scans the broader market news pool (not just ticker-filtered articles) for commodity/macro keywords (crude oil, rupee, gold, steel, natural gas, coal, sugar, aluminum). Direction inferred from article text (40+ up/down keywords). Per-ticker sensitivity: each ticker has its own direction (e.g., ONGC benefits from crude rise, OMCs are hurt). Ticker mention scanning limits results to tickers actually mentioned in articles. `focus_ticker` filters to only commodities affecting the searched ticker. Falls back to cautious Bearish when direction is unclear. No extra network calls.
7. **Indicators** — `indicators.py` computes RSI, SMA crossover, and MACD from 2-year OHLCV
8. **Render** — `render.render_dashboard()` assembles a dark-themed HTML dashboard rendered via `st.components.v1.html()`
9. **Vote** — Users rate signal accuracy (👍/👎). Votes update Beta posteriors for each source in `persistence.py`, which feeds back into step 4

### Source Weighting (Bayesian Calibration)

Each news source carries a confidence weight that self-calibrates via Beta-Binomial inference:

```
Weight = α / (α + β)
```

- **α** = correct votes for this source
- **β** = incorrect votes for this source
- Prior: hand-tuned (α₀, β₀) per source based on editorial quality
- Posterior updates on every 👍/👎 vote

The blended signal score is weighted by these posteriors:

```
signal = Σ(wᵢ · sᵢ) / Σ(wᵢ)
```

Where `wᵢ` is source weight and `sᵢ` is average compound score for that source.

After ~10–50 votes, weights converge from guesses to measurements.

---

## Quick Start

### Prerequisites

- Python 3.11+
- `pip` or `uv` (recommended for speed)

### Install & Run

```bash
git clone https://github.com/AshayK003/nse-sentiment-analyzer.git
cd nse-sentiment-analyzer

# Install (choose one)
pip install -r requirements.txt
# or
uv pip install -r requirements.txt

# Launch
streamlit run app.py
```

Open **http://localhost:8501** in your browser.

### Optional: Enable FinBERT

FinBERT offers ~15–20% better accuracy on financial text than VADER. Requires ~2GB of additional dependencies.

```bash
pip install torch transformers
USE_FINBERT=true streamlit run app.py
```

### Optional: FII/DII Institutional Data

```bash
pip install nsepython
```

The app works without it — this data appears as an informational section when available.

---

## Environment Variables

| Variable | Required | Purpose |
|----------|----------|---------|
| `USE_FINBERT` | No | Set to `true` to enable FinBERT transformer. Requires `transformers` + `torch` (see `requirements.txt`). |

**That's it.** No API keys, no tokens, no secrets. All data comes from free public sources.

---

## Project Structure

```
nse-sentiment-analyzer/
├── app.py                  # Streamlit entry point, UI state machine
├── data_fetcher.py         # yfinance, RSS (feedparser), DuckDuckGo search
├── sentiment.py            # VADER + financial lexicon, FinBERT, source weights
├── event_classifier.py     # 19 event types with signed sentiment bias
├── aggregate_sentiment.py  # SmartScore 0–100 (EWMA, breadth, volume, events)
├── cascade.py              # Cascade/ripple tracking — commodity → ticker effects
├── indicators.py           # RSI(14), SMA crossover, MACD
├── intraday.py             # VWAP, pivot levels, India VIX
├── market_data.py          # FII/DII flow, Market Mood Index (MMI)
├── persistence.py          # JSON file I/O, Bayesian source accuracy
├── render.py               # Dark-themed HTML/CSS dashboard + TradingView chart + overlays + a11y
├── requirements.txt
├── pyproject.toml          # Pytest config, coverage, markers
├── CHANGELOG.md            # Version history
├── LICENSE                 # AGPL v3
├── .streamlit/
│   └── config.toml         # Dark theme + minimal chrome
├── data/                   # Runtime data (gitignored)
│   ├── cache.json          # API response cache (15-min TTL)
│   ├── portfolio.json      # Saved tickers
│   ├── entry_prices.json   # P&L entry prices
│   ├── track_record.json   # Vote history
│   ├── source_accuracy.json # Bayesian posteriors per source
│   └── sentiment_history.csv # Daily SmartScore time series
├── .price_cache/           # L2 disk cache for price history (gitignored)
└── tests/
    ├── conftest.py         # Fixtures: tmp data dir, mock stock data
    ├── test_analyze_ticker.py    # Full pipeline end-to-end
    ├── test_data_fetcher.py
    ├── test_indicators.py
    ├── test_persistence.py
    ├── test_render.py
    ├── test_sentiment.py
    ├── test_event_classifier.py
    ├── test_cascade.py          # Commodity cascade/ripple detection
    ├── test_aggregate_sentiment.py
    ├── test_market_indicators.py  # Volume spike detection
    ├── test_entry_prices.py       # P&L tracking
    ├── test_intraday.py           # VWAP, pivots, VIX
    ├── test_changelog.py          # Infrastructure
    └── test_history_export.py     # CSV export
```

**10 source modules**, **14 test files**, **180 tests**.

---

## Local Development

### Adding a News Source

1. **Fetch function** — Add to `data_fetcher.py`. Return items as `{"title", "body", "url", "date", "source"}`
2. **Register prior** — Add source to `persistence.py:SOURCE_WEIGHTS_PRIOR` dict (used as Bayesian prior)
3. **Wire into pipeline** — Call your fetcher from `data_fetcher.py:search_news()`
4. **Test** — Add tests in `tests/test_data_fetcher.py`

### Adding a Technical Indicator

1. Add function to `indicators.py:get_technical_indicators()`
2. Add HTML display in `render.py:render_dashboard()`
3. Add tests in `tests/test_indicators.py`

### Code Conventions

- **Sync-first** — no async. Parallelism via `concurrent.futures.ThreadPoolExecutor`
- **Mock all external APIs** in tests — never hit production services
- **Use `cache_get`/`cache_set`** from `persistence.py` for API response caching
- **Prefer deletion over abstraction** — YAGNI as a principle. When in doubt, leave it out.
- **Lucide SVGs** for all UI icons — no emojis where an SVG serves the same purpose

---

## Testing

```bash
# Full suite (180 tests, mocked APIs, no network)
python -m pytest tests/ -v -q

# With coverage
python -m pytest tests/ --cov

# Specific file
python -m pytest tests/test_sentiment.py -v

# Specific test
python -m pytest tests/test_indicators.py::TestIndicators::test_rsi -v

# Slow tests (hit real APIs — not run by default)
python -m pytest tests/ -m slow

# Regression tests (previously fixed bugs)
python -m pytest tests/ -m regression
```

### Test Design

- **All external APIs mocked** — `yfinance`, `feedparser`, `duckduckgo_search`, `requests` patched with `pytest-mock`
- **Fixtures** in `conftest.py` provide isolated `tmp_data_dir` for file I/O + `sample_hist` DataFrame for indicators
- **Integration tests** verify the full pipeline at module boundaries (stock data → sentiment → event classification → SmartScore)
- **Markers** defined in `pyproject.toml`:
  - `slow` — tests hitting real APIs (opt-in)
  - `regression` — tests for previously-fixed bugs

---

## Deployment

### Streamlit Community Cloud (Free)

1. Fork/push to GitHub
2. Visit [streamlit.io/cloud](https://streamlit.io/cloud)
3. **New app** → select repo → branch → `app.py`
4. Deploy

The app runs at `https://<your-app>.streamlit.app`.

### Notes

- **Ephemeral filesystem** — portfolio, track records, and sentiment history are session-only on Streamlit Cloud. Data resets on each deploy.
- **RSS + DuckDuckGo** work on the cloud.
- **`nsepython`** is local-only (not available on Streamlit Cloud).
- **`yfinance`** can throttle if you queue many tickers rapidly. Use portfolio briefing mode for batch scans (skips news, ~1.5s/ticker).
- **No environment secrets needed** — all datasources are public APIs.

### Resetting Cached Data

```bash
# Clear local JSON cache
rm -f data/cache.json
```

Or click the cache stats button in the app sidebar.

---

## Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| `ModuleNotFoundError: nsepython` | Optional dep missing | `pip install nsepython` (app works without it) |
| yfinance returns nothing for a ticker | Delisted, suspended, or not on Yahoo Finance | Try `.NS` suffix manually via custom input |
| RSS feeds return empty | Rate-limiting or network issue | DuckDuckGo fallback kicks in automatically |
| Dashboard shows stale data | Cache TTL (15 min) hasn't expired | Clear cache via sidebar button, or wait |
| Streamlit Cloud "Module not found" | Missing dependency in `requirements.txt` | Add it and redeploy |
| Duplicate track record entries | Repeated searches created extra rows | Update to latest version (dedup is automatic) |

### Legal Disclaimer

This tool is for **educational and informational purposes only**. Nothing on this platform constitutes investment advice. The creator is not a SEBI-registered advisor. Data comes from third-party public APIs and may be delayed or inaccurate. Always consult a SEBI-registered financial advisor before making investment decisions.

---

## Contributing

See **[CONTRIBUTING.md](CONTRIBUTING.md)** for guidelines on PR workflow, testing requirements, and what we need/avoid.

---

## Support

If this tool saves you a bad trade or helps you learn, [buy the developer a chai](https://chai4.me/ashaykushwaha003).

---

## License

**GNU AGPL v3** — see [LICENSE](LICENSE).

This license ensures the code stays open and prevents closed-source monetization. Anyone who uses or modifies this code and runs it as a network service must release their changes under the same terms.

---

<div align="center">
  <sub>Built by <a href="https://x.com/sentinelcipher">@sentinelcipher</a></sub>
  <br/>
  <sub>Not financial advice. Always do your own research.</sub>
  <br/><br/>
  <img src="https://komarev.com/ghpvc/?username=AshayK003&repo=nse-sentiment-analyzer&style=flat&color=6b7280" alt="Profile Views"/>
</div>

nse-sentiment-analyzer

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README