Weak Signals Pipeline

A config-driven pipeline for detecting weak and emerging research signals from academic literature (e.g. arXiv) using term/document frequencies, Degree of Visibility (DoV), Degree of Diffusion (DoD), and signal maps (KEM/KIM). Outputs validated weak/strong signals and evolution visualizations.

---

Quick summary

The pipeline turns a corpus of papers (titles/abstracts) and a keyword list into signal maps and validated weak/strong signals over time, plus visualizations. It’s built to surface emerging or under-the-radar topics in a field—things that are gaining traction but aren’t yet dominant—so you’re not only looking at what’s already big.

Researchers and librarians can use it to scan a domain periodically and see which themes are rising or shifting. Strategy and foresight teams get a simple, reproducible way to monitor literature without manual tagging. You run it when you have a domain-specific corpus and a curated keyword set; it’s not a replacement for full-text search or citation analysis, just a complementary view on presence and growth of terms over time. Pipeline and config (periods, keyword list, etc.) control the outputs, so you can re-run for new years or new domains as needed.

---

Features

• Data ingestion: Filter raw JSON/JSONL by keyword and year range; output parquet.
• Metrics: Term frequency (TF), document frequency (DF), DoV, DoD with configurable time-weight w.
• Signal maps: Keyword Emergence Map (KEM) and Keyword Issue Map (KIM) with quadrant categorization (Strong / Weak / Latent / Well-known).
• Validation: Cross-validate KEM and KIM categories; export all validated and high-impact signals.
• Visualization: Plotly/Holoviews/Bokeh plots (yearly keywords, innovation rate, Jaccard overlap, Sankey, treemap, heatmap).
• Notebooks: Exploratory analysis and embedding-based keyword extraction (sentence-transformers).
• Config-first: All paths and parameters in config/config.yaml; no hardcoded machine paths.

---

Requirements

• Python 3.9+
• Data: ArXiv-style JSON/JSONL (fields: id, update_date, title, abstract).
• Keywords: A text file with one standardized keyword per line (used for TF/DF).

---

Installation

From the project root:

# Clone (if needed)
git clone https://github.com/sidsharmaa/weak-signals-new.git
cd weak-signals-new

# Virtual environment (recommended)
python -m venv venv
# Windows: venv\Scripts\activate
# macOS/Linux: source venv/bin/activate

# Install in editable mode so `src` is importable
pip install -e .
pip install -r requirements.txt

Verify:

python test_environment.py

---

Configuration

Edit config/config.yaml before running the pipeline.

Section	Purpose
data	raw_path, processed_path, external_keywords (path to keyword list).
metrics	Output paths for TF, DF, DoV, DoD CSVs.
processing	keywords (filter for raw data), start_year / end_year, optimal_w, dov_w_value_to_test.
analysis	periods (p1/p2/p3 year ranges), reports_dir, KEM/KIM evolution paths.
validation	Paths for all-validated and high-impact signal CSVs.
visualization	yearly_keywords_path, output_dir for HTML/plots.

Paths in the config are relative to the project root. The app uses a single config and get_project_root() for resolution.

---

Pipeline Usage

Run from the project root (with venv activated). Order matters: later steps depend on outputs of earlier ones.

1. Build dataset (raw → processed)

Filters raw JSON by config keywords and year range; writes parquet.

python -m src.data.make_dataset

2. Term frequency (TF)

Requires: data/external/<keyword file>, processed parquet.

python -m src.scripts.run_tf_calculation

3. (Optional) Find optimal DoV weight w

Tests dov_w_value_to_test and can update optimal_w in config.yaml.

python -m src.scripts.run_w_experiment

4. Degree of Visibility (DoV)

Requires: TF CSV, processed parquet.

python -m src.scripts.run_dov_calculation

5. Document frequency (DF) and Degree of Diffusion (DoD)

Requires: keyword file, processed parquet. Writes DF then DoD using optimal_w.

python -m src.scripts.run_df_dod_calculation

6. KEM analysis (Keyword Emergence Map)

Requires: TF CSV, DoV CSV. Writes period CSVs, plots, and evolution CSV.

python -m src.scripts.run_kem_analysis

7. KIM analysis (Keyword Issue Map)

Requires: DF CSV, DoD CSV. Writes period CSVs, plots, and evolution CSV.

python -m src.scripts.run_kim_analysis

8. Signal validation

Requires: KEM and KIM evolution CSVs. Writes all-validated and high-impact signal CSVs.

python -m src.scripts.run_signal_validation

9. Yearly keyword visualizations

Requires: reports/yearly_final_keywords.txt (or path set in config.visualization.yearly_keywords_path). Writes HTML to config.visualization.output_dir.

python -m src.visualization.visualize

Optional visualizations

• Streamgraph (heatmap): python -m src.visualization.streamgraph
• Sankey (signal flow): python -m src.visualization.sankey
• Treemap (P2→P3): python -m src.visualization.evolutionary_pathway_treemap
• Holoviews/Bokeh: python -m src.visualization.visualizenew

These read from config (e.g. validation.high_impact_path, metrics.dov_path, visualization.yearly_keywords_path).

---

Project Structure

weak-signals-new/
├── config/
│   └── config.yaml          # Single source of paths and parameters
├── data/
│   ├── raw/                 # Raw input (e.g. arxiv JSON/JSONL)
│   ├── processed/           # Filtered parquet and derived CSVs (TF, DF, DoV, DoD)
│   └── external/            # Keyword list and other external inputs
├── docs/
├── models/                  # Saved models (optional; placeholders in src)
├── notebooks/               # Jupyter notebooks (exploration, embedding keywords)
├── reports/                 # Output reports and figures
│   └── figures/             # KEM/KIM plots, evolution CSVs, validated signals
├── src/
│   ├── analysis/            # Signal map logic (KEM/KIM)
│   │   └── signal_maps.py
│   ├── config.py            # Config load and project root
│   ├── data/                # Data ingestion
│   │   ├── data_ingestion.py
│   │   └── make_dataset.py
│   ├── features/            # Embedding-based keyword extraction (notebooks)
│   │   └── build_features.py
│   ├── models/              # Train/predict placeholders
│   │   ├── train_model.py
│   │   └── predict_model.py
│   ├── processing/          # TF, DF, DoV, DoD
│   │   └── metrics.py
│   ├── scripts/             # Pipeline entrypoints (run_*.py)
│   ├── tests/
│   │   └── test_signal_maps.py
│   ├── utils/
│   │   └── common.py        # NLTK resource setup
│   └── visualization/       # Plotting and HTML export
│       ├── visualize.py
│       ├── streamgraph.py
│       ├── sankey.py
│       ├── evolutionary_pathway_treemap.py
│       └── visualizenew.py
├── Makefile
├── requirements.txt
├── setup.py
└── test_environment.py

---

Development

• Tests: pytest src/tests/
• Lint: flake8 src (or make lint where supported)
• Environment check: python test_environment.py

Notebooks in notebooks/ use src.features.build_features (embedding model, keyword normalization) and, in one case, sparse TF/DF helpers; run with the project root on PYTHONPATH or after pip install -e ..

---