Browser4 Web Miner

WebMiner — Convert Detail Web Pages into Interactive Views

Quick Start

bashCopy code
browser4-cli webminer install            # one-time install (Java 17+ auto-detected)
browser4-cli webminer all <html-dir>     # full pipeline: encode → cluster → views

WebMiner groups similar web pages together and produces an interactive HTML report with clusters of related pages — plus Excel spreadsheets for further analysis. Give it a folder of downloaded HTML files, and it handles the rest. Everything runs locally; no data leaves your machine.

When to Use

Use WebMiner when you have a folder of downloaded HTML pages and want to cluster them into interactive views and Excel reports — fully local, no LLM tokens. It complements rather than replaces browser4-cli crawl/swarm (which acquire pages): WebMiner analyzes pages you already have. Not for single-page extraction — use htmlsnapshot for that.

How It Works

WebMiner runs a three-stage local pipeline: encode converts each HTML page into a 69-dimension feature vector, cluster groups similar pages with SMILE KMeans (k auto-detected), and views renders an interactive HTML report plus Excel spreadsheets. Everything runs locally on your machine — no data leaves it, and no LLM tokens are consumed.

Patterns

1. Full pipeline on a folder of pages

bashCopy code
browser4-cli webminer all <html-dir>

2. Rebuild views from an existing run

bashCopy code
browser4-cli webminer views <result-dir>

3. Try it on the sample dataset

bashCopy code
browser4-cli webminer run-example

Flags

Errors & Recovery

Using from the Browser4 CLI

WebMiner is a first-class Browser4 citizen: the browser4-cli webminer command installs, updates, and runs the tool natively (no PowerShell needed — the CLI locates a Java 17+ installation, preferring the JRE bundled with the Browser4 runtime, and launches scent-miner.jar directly). The JAR and its release metadata are installed to ~/.scent/webminer/.

bashCopy code
browser4-cli webminer install            # Download and install the latest release
browser4-cli webminer update             # Check for and install the latest release
browser4-cli webminer version            # Show installed and latest available versions
browser4-cli webminer uninstall          # Remove the installed release
browser4-cli webminer run-example        # Sample dataset + full pipeline (needs 7-Zip)
browser4-cli webminer all <html-dir>     # Full pipeline (encode → cluster → views)
browser4-cli webminer views <result-dir> # Rebuild views from an existing run

• webminer all <dir> accepts the pipeline options directly (--max-files <n>, --output <dir>, --resume [<project-id>]).
• Any other command is forwarded verbatim to scent-miner.jar, e.g. browser4-cli webminer encode <dir>.
• Runs started through the CLI set -Dapp.name=webminer, so the views task-output root is %TEMP%\webminer-<user>\ml\tasks\... (<user> is the OS user name; see Output).
• The bare webminer panel and webminer version keep the update check quiet: the GitHub → OSS-mirror fallback notices (rate limit, HTTP status, unreachable) are suppressed, and the Published line is omitted entirely when the release carries no published_at. webminer install / update still report the fallback.

Installing WebMiner

browser4-cli webminer install downloads, verifies, and installs the latest release (GitHub Releases with an Aliyun OSS mirror fallback; works on Windows, Linux, and macOS — no PowerShell needed):

bashCopy code
browser4-cli webminer install            # Download and install the latest release
browser4-cli webminer update             # Check for and install the latest release
browser4-cli webminer version            # Show installed and latest available versions
browser4-cli webminer uninstall          # Remove the installed release

Releases are installed to ~/.scent/webminer/ and checked against https://github.com/platonai/web-miner/releases. SHA-256 checksums are verified automatically on download.

You can also use the JAR directly if it's already available:

bashCopy code
java -jar scent-miner.jar <command> <args>

Converting Pages to Views

Running the Example

The run-example command downloads a pre-uploaded test dataset of real web pages, extracts it, and runs the full pipeline — no manual setup required beyond Java 17 and 7-Zip:

bashCopy code
browser4-cli webminer run-example

The dataset is cached at ~/.scent/test-data/amazon.com/ so subsequent runs skip the download.

Running on Your Own Pages

bashCopy code
# Full pipeline (one-shot)
browser4-cli webminer all /path/to/html/files

# Or with the JAR directly
java -jar scent-miner.jar all /path/to/html/files

The cluster count is always auto-detected from the data — this produces better results than guessing a number.

Options

Building Views from an Existing Run

If clustering has already completed and you just need to (re)build the views:

bashCopy code
java -jar scent-miner.jar views <html-dir>-ml-output/kmeans-result/p<timestamp>

Output

all produces two kinds of artifacts in two different places:

1. Clustered results — written to <html-dir>-ml-output/kmeans-result/p<timestamp>/ (or wherever --output points): one result.csv per feature view (predictionAnd{Final,Minimal,Original}Features/result.csv) plus clusteringInfo.txt.
2. Views (interactive HTML report + Excel + JSON) — the views stage of all writes them to the application's temp task-output root, NOT under <html-dir>-ml-output: %TEMP%\<app>-<user>\ml\tasks\unsupervised\result\p<timestamp>\predictionAndMinimalFeatures.views\ on Windows, and <java.io.tmpdir>/<app>-<user>/ml/tasks/unsupervised/result/p<timestamp>/predictionAndMinimalFeatures.views/ on Linux/macOS (/tmp/... on Linux, $TMPDIR on macOS) — the <app> prefix follows -Dapp.name (webminer when launched through browser4-cli webminer, pulsar for a direct java -jar run) and <user> is the OS user name. The end of the run prints the resolved absolute views path.

So after java -jar scent-miner.jar all ./html-pages/ the clustered results look like:

Copy code
html-pages-ml-output/
  └── kmeans-result/
      └── p<timestamp>/
          ├── predictionAndFinalFeatures/result.csv
          ├── predictionAndMinimalFeatures/result.csv
          ├── predictionAndOriginalFeatures/result.csv
          └── clusteringInfo.txt

and the views (<project>.html, *.xlsx, *.json) live in the temp task-output directory printed by the run.

The real report is <project>.html (e.g. p<timestamp>.html), not index.html. The index.html inside the views directory is an auto-generated directory listing ("Index of predictionAndMinimalFeatures.views") — opening it shows a file list, not the interactive clustering report. Open <project>.html instead.

To place the views beside the clustered results (e.g. to archive them with the project), rebuild them from the result directory:

bashCopy code
browser4-cli webminer views <html-dir>-ml-output/kmeans-result/p<timestamp>
# (equivalent to: java -jar scent-miner.jar views <html-dir>-ml-output/kmeans-result/p<timestamp>)

This writes predictionAndMinimalFeatures.views/ inside the given result directory — the recommended way to locate artifacts, since the output path is explicit instead of an opaque temp path. Open the generated <project>.html in a browser to explore the clustering results. The .xlsx files can be opened in Excel for sorting, filtering, or further analysis.

Tips

• Input files — only *.html and *.htm files are processed. Other files in the directory are ignored.
• Resume interrupted runs — if a pipeline stops partway through, use --resume to continue from the last completed stage instead of starting over.
• Offline only — WebMiner works with pre-downloaded HTML files. Use a browser, wget, or a crawler to fetch pages first.
• Java 17 is required. Make sure java is on your PATH.