Unified workflow
Listings To Scraper Workflow
One route map for collection, scrape operations, normalization, pipeline replay, health, versions, and vocabulary ownership. Legacy extract/trainer/errors/admin deep links now open the in-console Normalization workspace.
ListingsListings ConsoleFilter, rank, inspect, and drill into collected profiles.Profile ResultsListings ConsoleNo-code match/exclude criteria with live, mobile profile cards.ScannerListings ConsoleConfigure scan launch inputs and runtime compatibility.Scraper RunsScraper AdminStart, stop, monitor, replay, and inspect crawler output.NormalizationScraper AdminReview samples, patterns, candidates, trainer, and normalization errors.PipelineScraper AdminReplay, rematerialize, audit, and resolve queue exceptions.Map/NormalizeScraper AdminMap fields, preview output, normalize profiles, and publish versions.HealthScraper AdminCheck site health, provenance, trace correlation, and drift.VocabularyTag StudioOwn domains, tags, groups, rankings, filter groups, and table contracts.
Listings Workspace Studio
Local layouts + quick launches
Saved layouts are browser-local workspace presets for this Listings console only. Use Tag Studio Table Views for canonical shared presets, then launch rankings, domains, tags, groups, and filter chips from Tag Studio-driven config. Button groups are drag-reorderable and persisted locally without changing raw data.
Normalization
Idle
Waiting for the next extraction run.
Listings Runtime Tools
Idle
Waiting for the next runtime action.
Runtime Report
Listings Table Controls
Table Ribbon
Column Colors
Listings column color settings
Use this slide-out panel to tune the Listings column highlights. The live header rules stay bound to the metadata family, so the notes update automatically if the data source changes.
Yellow highlight
Scraped profile headers
Forest green highlight
Tag Studio registry-backed headers
Header note visibility
Show source notes in the header row
Highlight strength
97%
Move the slider to make the header highlight lighter or stronger. The color rules still follow the live column metadata family.
Profile Normalization
Listings projection readiness
Projection surface
Listings is a fast projection surface. Effective normalized profile values, override versions, rollback, and provenance stay owned by scraper-mapping; this drawer shows the safe handoff and row readiness without direct DB writes.
Listings Table Controls
Table Ribbon
Grouped table controls for layout, structure, views, filters, status, utility, and a far-right workspace collapse action.
Visible Rows
Loading
Raw Rows Loaded
Loading
Default Load
48h window
Run Start
Pending
Run End
Pending
Source
Pending
Filters
Filter Groups
Match
Exclude
Effective Rules Enabled:
No match or exclude terms are active.
| Loading... |
| Name | Phone | Site | x.Services | Title | Location | Price | Posted | Age | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Loading... | ||||||||||||||
Scanner
The scraper crawls configured escort sites with Playwright and extracts listings. Raw output goes to
scrape-results.json which feeds the Listings tab. Site Configuration now lives on this same Scanner surface so the site registry, session health, and run controls stay in one place. The launch controls here intentionally stay on the canonical API contract only: sites, mode, max pages, and unattended execution.
Terms:
Site = one escort site (megapersonals, tryst, listcrawler)
Mode = normal (full drill) / fast (listings only)
Max Pages = per-site cap
Unattended = suppress prompts
Steps:
1 Configure sites and flags
2 Review generated API request
3 Start / stop scan
4 Watch live log and health
5 Edit runtime families and flat keys (CRUD)
6 Validate output on Listings
Scanner now hosts full runtime administration. The control plane below exposes live family inventory, family-level JSON CRUD, and reset operations so runtime changes happen here without handoff. Scraper Admin links remain available for replay/provenance operations.
Add Site
Branding & Metadata
Runtime Admin Control Plane
Loading runtime families…
Step A: review live family inventory
Step B: choose family edit target
Step C: apply family JSON CRUD updates
Step D: manage flat key CRUD
Step E: save/reset and validate readback
| Family | Keys / Flags | Backend | Action |
|---|---|---|---|
| Loading runtime families… | |||
Family JSON CRUD
Flat Key Inventory
| Key | Preview |
|---|---|
| Loading flat keys… | |
Flat Key CRUD
Compat History & Rollback
Site Probe (Compat)
Mounting runtime admin editor…
Live log
Run history
Loading…
Normalization handoff
Loading…
Site health
Loading…
Add Subtab
📹 Scrape
IdleBridge surface. Use Scraper Admin as the primary runtime/operator console for start/stop, live log, per-site health, diagnostics, replay, and deeper semantic operations. Snapshot browsing now belongs on Snapshot Center. Listings Scrape remains a compact launch/status bridge and not the primary admin owner surface.
Scan Launcher and Runtime Status
Start or stop a scrape run and watch the current runtime state update live.
Live status
LoadingScraper Runtime
Hydrating from canonical runtime sources...
Runtime status-
Running now-
Last run-
Next run-
Last duration-
Status source-
Waiting for runtime status...
Live Run Log
Streams the active scraper log tail while this Scrape route is open.
Streams the active scraper log tail while this page is open.
Waiting for scraper log output...
Tag Distribution and Site Health
Shows current tag counts and the recent per-site health summary for monitored domains.
Refreshes tags and health from the live scraper APIs.
Current Tag Distribution
Loading...
Per-Site Health Summary
Merges Sites Registry records with recent scraper history so every configured site shows current telemetry or an explicit no-data explanation.
| Site | Runs | Avg Results | Error Rate | Last Run | Latest Status | Detail / Remediation |
|---|---|---|---|---|---|---|
| Loading... | ||||||
Recent Runs and Diagnostic Summary
Review recent runs with failure details, remediation guidance, and the latest synthesized troubleshooting snapshot.
Recent Run History
| Started | Duration | Sites | Mode | Results | Status | Diagnostic / Remediation |
|---|---|---|---|---|---|---|
| No history available. | ||||||
Last Scan Diagnostic Snapshot
Synthesizes the latest stored status, recent run history, and diagnostic log tail into one operator-focused troubleshooting snapshot.
- Status
- Loading...
- Last Run
- —
- Results
- —
- Sites
- —
- Failure Class
- —
- Description
- —
- Remediation
- —
- Log File
- —
If the latest run has no site-level telemetry yet, this panel will tell you whether the gap came from an API restart, no completed run, or missing site attribution in stored history.
Keyword and Filter Preview Tools
Test keyword matching and preview filter impact before applying retag operations.
Keyword Tester
Enter a listing description to see which filter tags would be applied.
Filter Impact Preview
Storage-only dry-run retag against the current live rows and registry-backed policy. No writes occur; this preview reports before/after tag deltas and sample changed rows.
—
Total Runs
—
Listings Processed
—
Avg / Run
—
Error Rate
Listings / Run (trend)
Tag Distribution
Errors by Site
Recent Runs
| Run ID / Time | Sites | Listings | Errors | Duration | Status |
|---|---|---|---|---|---|
| Loading… | |||||
The editor below is the live runtime/admin control surface for scraper configuration on this page. Every family is loaded from the current API state, and edits save back through the owning backend so admins have real create/update/delete control instead of a read-only mirror.
Runtime Catalog
--
Families
--
Flat Keys
--
Uncovered
--
System Rows
| Family | Owner | Live Preview |
|---|---|---|
| Loading runtime configuration catalog... | ||
Full Compatibility Snapshot
Raw live snapshot of the effective scraper compatibility payload. This is the seam-of-last-resort editor when an admin needs to touch every key directly, including compatibility-only values.
Family Editor
Select a family from the runtime catalog to inspect the live payload and edit it directly.
Admin Notes
- Family editors are live JSON CRUD surfaces: add objects, update properties, or delete rows/keys directly in the payload and click Save Family.
- Reset resolves through the owning API or canonical runtime defaults. The fallback bootstrap file stays server-side and is no longer surfaced in this admin console.
- The catalog shows the grouped domain/owner view, while the full compatibility snapshot preserves seam-level visibility across every flat key exposed by the runtime.
- Scraper Admin remains the right place for patterns, replay, provenance, queueing, and deterministic semantic workflow; this tab is for runtime configuration control.
Pipeline
Visual order-of-operations editor. Reads and writes scan_pipeline via the scan runtime API. Drag stages to reorder. Click a stage to configure its settings. For the full Training Workbench rail (Samples -> Patterns -> Candidates -> Map -> Preview -> Normalize -> Publish -> Replay -> Monitor), use Scraper Admin.
Training Workbench
Samples -> Patterns -> Candidates -> Map -> Preview -> Normalize -> Publish -> Replay -> Monitor
Use this rail when onboarding or repairing a site: sample deterministic snapshots first, inspect pattern reasons, verify candidate evidence, map to Tag Studio-owned vocabulary, preview, review normalized profile output, publish, replay, and monitor.
Before Scan
→
Scan + Exclude Always
→
Post-Scan
→
Output
Runtime: scanner
| Step: -
| Status: -
Log
No site schemas yet. Add a site above to start building one.
Detail Page Alignment Workspace
Map site detail data into the canonical Listings contract.
Use a saved HTML detail page or a live listing URL, inspect what the normalizer sees, then decide which canonical field and Listings column contract should own each value. This keeps detail-page parsing, registry ownership, and Listings rendering aligned instead of drifting independently.
0Normalized fields in current sample
0Listings mappings currently enabled
0Schema rules active for this site
Semantic review/publish and runtime CRUD now route through the single Scraper Admin owner for deterministic workflow controls. Normalization keeps schema, trainer, and parse diagnostics local; canonical mapping edits live in the Tag Studio domain owner surface.
Mandatory Design Note
All current Magnetar processes must run with Windows-native mechanisms and solutions. Keep portability seams clean for the future Synology NAS migration, but do not design new flows that depend on Linux-only assumptions during the current Windows phase. The preview below reads the canonical scraper.attribute_display_map contract, while editing lives in the Tag Studio domain owner surface.
Detail Page Source
Saved HTML
Live URL
Provider
Loading...
File
Select a provider first
📊 All Sites → Schema Coverage
shows all configured sites and which fields have schema rules
▼ collapse
Click header to load coverage matrix
Select a site-backed domain on the left to edit normalization rules. If the list is empty, run a scrape first or seed a schema in the Schema Manager tab.
Domain
Domains come from the configured Sites registry. Placeholder/example domains are hidden and cannot be saved from this pane.
Choose a configured site domain to start field-level schema authoring.
Loading schemas...
No domain selected
Add a field key for the selected domain, then edit one normalization rule per line below.
Select a domain, then add a field key like age or nationality.
Select a domain to edit its schema.
Test Normalization
No result files found in
Z:\\magnetar\\web_scraper\\results\\. Run a scrape from Scan → Scanner to generate one.
⚠ Parse Error Log
per-domain extraction errors from recent scrape results
Last refreshed: never
Select a result file above to load parse errors.
Single admin owner: Normalization no longer renders a second runtime CRUD console. Runtime families, compatibility keys, static controls, versions, replay, and publish actions live in Scraper Admin so operators have one authoritative admin site.
Normalization Admin Collapsed
Use Scraper Admin for runtime control.
This panel is a compatibility handoff only. It preserves the old Normalization Admin landing area for keyboard/local fallback flows while keeping trainer, parse-error, and admin deep links stable inside the console.