Results Export For Web of Science

简体中文 | 繁體中文

Export Web of Science search results in large batches, then convert the raw WoS field-tagged files into clean CSV, TSV, JSONL, or merged WoS TXT outputs.

This fork keeps the original Go exporter and adds a local Web UI, direct WoS URL support, managed output folders, selectable columns, conservative Crossref DOI filling, and an automatic SID helper for modern Web of Science sessions.

Web UI Preview

What This Fork Adds

• Local Web UI: paste a WoS summary URL, choose record range, columns, format, output folder, concurrency, and resume behavior.
• Direct URL workflow: no need to manually extract the QID in normal use.
• Automatic SID helper: first tries to read a valid Web of Science SID from local Chrome, Edge, or Brave browser storage, then validates it against WoS before using it.
• Manual SID fallback: paste X-1P-WOS-SID from the browser Network panel if automatic detection fails.
• Managed outputs: exports are stored under exports/wos_<timestamp>_<qid-prefix> unless you name the task folder.
• Multiple output formats: CSV, TSV, JSONL, and raw merged WoS TXT.
• Selectable fields: defaults to authors,journal,year,doi,title, with additional metadata fields available in the UI.
• Crossref DOI filler: fills blank DOI cells with conservative title, journal, year, and first-author checks.
• cURL fallback exporter: reuse an authenticated browser saveToFile request when session handling is difficult.

Requirements

• Legal access to Web of Science through your institution or account.
• Windows is the tested environment for the Web UI launcher.
• Python 3.10+ for the Web UI and Python CLI tools. The core Web UI uses only the Python standard library.
• Go is optional, only needed if you want to build or run the original Go tools.
• For automatic SID detection, keep a logged-in WoS page open in Chrome, Edge, or Brave on the same machine.

Quick Start: Web UI

On Windows, double-click:

start_web_ui.cmd

The script creates .venv if needed, activates it, starts the local server, and opens the browser.

Manual start:

cd E:\EK_CODE\ResultsExportForWebOfScience
python .\wos_web_ui.py --host 127.0.0.1 --port 8765 --open

If port 8765 is occupied, use another one:

python .\wos_web_ui.py --host 127.0.0.1 --port 8766 --open

Then:

1. Search in Web of Science.

2. Copy the current WoS summary URL, for example:

   https://webofscience.clarivate.cn/wos/woscc/summary/<QID>/<route-id>/relevance/1
   

3. Paste it into the Web UI.

4. Enter the total record count shown by WoS.

5. Choose record range, output format, columns, output root, task folder, and concurrency.

6. Start export.

Split files are saved as:

000001_001000.txt
001001_002000.txt
...

The final selected-field file is saved as:

wos_selected_fields.csv
wos_selected_fields.tsv
wos_selected_fields.jsonl

For raw WoS TXT output, the chunks are merged into:

wos_selected_fields.txt

SID Handling

Modern Web of Science does not always put the export SID in the URL. The second UUID-like path segment in a summary URL is often only a page route ID, not the X-1P-WOS-SID accepted by the export API.

This fork handles SID in this order:

1. Use the manually entered SID if provided.
2. Use a URL segment only if it looks like a real export SID, not a UUID route ID.
3. Try to read SID candidates from local Chrome, Edge, or Brave WoS storage.
4. Validate each candidate with POST /api/esti/Session/getSessionData.
5. Fall back to the original Web of Knowledge redirect method.

If all automatic methods fail:

1. Open browser developer tools.
2. Go to the Network tab.
3. Trigger a WoS export or locate a saveToFile request.
4. Copy the request header X-1P-WOS-SID.
5. Paste it into the Web UI SID field.

Do not publish your SID or copied cURL commands. They can contain live session credentials.

Output Options

Supported formats:

Format	Description
CSV	Excel-friendly UTF-8 with BOM
TSV	Tab-separated text
JSONL	One JSON object per record
WoS TXT	Raw merged WoS field-tagged text

Default columns:

authors,journal,year,doi,title

Additional selectable columns:

author_full_names,document_type,language,abstract,author_keywords,keywords_plus,
wos_id,issn,eissn,volume,issue,pages,article_number,times_cited

Export Limits

• One WoS export API request exports at most 1,000 records. This tool automatically splits larger jobs into 1,000-record chunks.
• You can enter large totals such as 46223; the output will be split into files like 000001_001000.txt, 001001_002000.txt, and so on.
• The practical total limit is imposed by the Web of Science web interface, not by this repository. Historically, records beyond 100,000 may not be exportable from the web interface.
• If a search result set is larger than 100,000, split the query by year, document type, category, database edition, or another stable filter, then merge the exported files.

Supported Exported Content

CSV, TSV, and JSONL outputs can currently select these structured columns:

Column key	Content
authors	Authors from AU, falling back to AF
author_full_names	Full author names
journal	Source title, journal, book, or proceedings title
year	Publication year
doi	DOI
title	Record title
document_type	Document type
language	Language
abstract	Abstract
author_keywords	Author keywords
keywords_plus	Keywords Plus
wos_id	Web of Science accession number / UT
issn	ISSN
eissn	eISSN
volume	Volume
issue	Issue
pages	Beginning and ending pages
article_number	Article number
times_cited	Times cited

When WoS TXT is selected, the raw WoS field-tagged text is preserved. The export request currently asks Web of Science for these fields:

WoS export field	Content
AUTHORS	Authors
TITLE	Title
SOURCE	Source, journal, book, or proceedings
DOI	DOI
CONFERENCE_INFO	Conference information
CITTIMES	Times cited
ACCESSION_NUM	Web of Science accession number
AUTHORSIDENTIFIERS	Author identifiers, such as ResearcherID or ORCID when available
ISSN	ISSN / eISSN information
PMID	PubMed ID
ABSTRACT	Abstract
ADDRS	Addresses
AFFILIATIONS	Affiliations
DOCTYPE	Document type
KEYWORDS	Keywords
JCR_CATEGORY	JCR category
SUBJECT_CATEGORY	Subject category
WOS_EDITIONS	Web of Science editions / indexes
CITREF	Cited references
CITREFC	Cited reference count
USAGEIND	Usage indicators
HOT_PAPER	Hot Paper flag
HIGHLY_CITED	Highly Cited Paper flag
FUNDING	Funding information
PUBINFO	Publication information
OPEN_ACCESS	Open access information
PAGEC	Page count
SABBR	Source abbreviation
IDS	IDS number
LANG	Language

Actual returned values depend on the record type, database edition, subscription access, and whether WoS has that metadata for a record.

Resume and Overwrite

The Web UI downloads WoS results in 1000-record chunks.

• Leave Overwrite existing content unchecked to resume an interrupted export. Existing non-empty chunk files are skipped.
• Check it to redownload and overwrite existing chunk files.

If the task status becomes Partial, WoS returned fewer records than requested. For example, requesting 1-13 may return only 10 field-tagged records if the current WoS session has only loaded/cached the first result page. Open the later result page in WoS, increase the result-page size if possible, or use the cURL fallback after triggering export in the browser.

Command Line Usage

Direct WoS URL export:

python .\wos_export.py "<WoS summary URL>" 46223 -o .\exports\my_search --workers 6

Manual SID override:

python .\wos_export.py "<WoS summary URL>" 46223 -o .\exports\my_search --sid "<X-1P-WOS-SID>" --workers 6

Extract the default five columns from raw WoS split files:

python .\extract_wos_fields.py .\exports\my_search -o .\exports\my_search\wos_selected_fields.csv

TSV output:

python .\extract_wos_fields.py .\exports\my_search -o .\exports\my_search\wos_selected_fields.tsv --format tsv

Crossref DOI Filling

After exporting a CSV with a doi column, fill missing DOI values conservatively:

python .\fill_missing_doi_crossref.py .\exports\my_search\wos_selected_fields.csv `
  -o .\exports\my_search\wos_selected_fields_crossref_filled.csv `
  --report .\exports\my_search\crossref_doi_report.csv `
  --cache .\exports\my_search\crossref_cache.json `
  --workers 10 `
  --mailto you@example.com

The filler only accepts matches that pass strict title similarity, token overlap, journal/year checks, and first-author checks where available. Review crossref_doi_report.csv before relying on filled values.

Troubleshooting Partial Exports

Partial means the tool requested a range, but the WoS export response contained fewer field-tagged records than expected. The output file is still written, but it is not complete for the requested range.

Example log:

Saved 1-13 (10/13 records)
Warning: WoS returned 10 records for requested range 1-13; expected 13.
Partial export: requested 13 records, but parsed 10.

This is usually not a parser problem. It means WoS did not provide all record UIDs to the export API for that session/range.

Try these fixes:

1. Open the same result set in WoS and browse to the missing page/range.
2. Increase the WoS results-per-page setting, if available.
3. Trigger WoS's own export dialog once, then retry in this tool.
4. Use the cURL fallback after copying a fresh authenticated saveToFile request.
5. Lower concurrency if larger exports return intermittent timeouts or session errors.

cURL Fallback

If automatic SID detection is not enough, copy an authenticated browser request:

1. Open WoS in your browser.
2. Open developer tools, Network tab.
3. Trigger an export or find saveToFile.
4. Right-click the request and copy as cURL.
5. Save it to wos_curl.txt.

Run:

python .\wos_export_from_curl.py .\wos_curl.txt 46223 -o .\exports\my_search --workers 4

This is useful when institutional login, proxy, or session behavior makes the normal URL workflow fail.

Original Go Tools

Run the Go exporter:

go run .\WebOfScienceResultsExport.go

Run the Go splicer:

go run .\WebOfScienceSplice.go

Build Windows executables:

go build -o WOSResultsExportTool.exe .\WebOfScienceResultsExport.go
go build -o WOSExportSpliceTool.exe .\WebOfScienceSplice.go

Notes and Limits

• Web of Science itself may limit exportable result positions. Historically, records beyond 100,000 may not be exportable from the web interface.
• Some WoS result sets are lazy-loaded. If WoS has not loaded/cached all record UIDs for the requested range, the export API may return fewer records than requested. The Web UI marks this as Partial.
• Large exports can take time. Lower concurrency if WoS returns timeouts or session errors.
• The local Web UI binds to 127.0.0.1 by default.
• The browser SID helper reads local browser storage only to find and validate WoS SID candidates. SID values are masked in logs.

Files

File	Purpose
start_web_ui.cmd	Windows one-click Web UI launcher
wos_web_ui.py	Local Web UI server
web_ui/	Frontend assets
wos_export.py	URL-based Python exporter
wos_export_from_curl.py	cURL-based fallback exporter
extract_wos_fields.py	Parse WoS TXT chunks into CSV/TSV
fill_missing_doi_crossref.py	Fill blank DOI cells with Crossref
WebOfScienceResultsExport.go	Original Go exporter, with URL/SID improvements
WebOfScienceSplice.go	Original Go splicer