DEV-20451: accurate per-pageview Time-on-Page metric

[caddoo]

Summary

• Replaces the derived "delta to next log_link_visit_action row" time-on-page calculation with a dedicated per-pageview row in a new log_page_view_time table, keyed on (idvisit, idpageview) so each tab (pv_id) accumulates its own time.
• Events, heartbeats, and content interactions now update the tab's own row instead of stealing time from the following pageview; the visit's final pageview is backfilled from visit_last_action_time at archive time instead of defaulting to 0 s.
• New Tracker.record_accurate_page_view_time = 1 kill-switch (temporary, planned for removal in Matomo 6.0). The archiver also probes log_page_view_time for the current archive period and falls back to the legacy time_spent_ref_action query when no rows exist — so mixed-schema states (mid-week upgrade, invalidation of pre-upgrade days) never silently collapse sum_time_spent to zero.
• New nb_hits_with_time_spent metric mirrors the existing nb_hits_with_time_generation convention; AverageTimeOnPage divides by it when present and falls back to nb_hits on pre-upgrade archives.

Related tickets

• DEV-20451 (this ticket): backend implementation of the accurate per-pageview time-spent metric.
• DEV-20460 (parent): "Time Spent on Page" improvement track.

Reference

PoC branch: dev-15222. This PR ports the PoC minus the pv_time client-time tracker parameter (to be handled in a follow-up ticket) and adds a presence-check safeguard in the archiver so re-archiving of pre-upgrade days stays on the legacy metric.

Performance

Two write layers add work on the hot tracker path: the previous-pageview close on a new TYPE_PAGE_URL, and the same-tab UPDATE for every follow-up hit (event, heartbeat, content, search). Both were measured on a seeded dataset of ~4 M log_page_view_time rows across ~500 k visits (single MySQL 5.7 instance, no tuning). Full report + EXPLAIN output at /tmp/dev20451-perf/REPORT.md locally; the highlights:

Per-hit tracker cost

Query	Access plan	Warm p50	Warm p95	Cold
Close previous pageview (ORDER BY id DESC LIMIT 1)	Range on index_idvisit_idaction_url, 1 row	6 ms	21 ms	21 ms
Same-tab UPDATE (WHERE idvisit=? AND idpageview=?)	Unique-key hit on unique_idvisit_idpageview	1.4 ms	30 ms	63 ms
INSERT ... ON DUPLICATE KEY UPDATE	Single-statement upsert, unique-key resolution	~1 ms	–	–

Every writer query is a single-row indexed operation. There are no full scans, no SELECT inside the writer path, and no session variables. The visit_standard_length cap is applied inline via LEAST(cap, GREATEST(time_spent, elapsed)) so a stalled tab cannot inflate a row beyond the visit cap.

Archive-time cost

The accurate archiver examines ~37 % fewer rows per day than the legacy path (1.94 M vs 3.06 M on the seeded day), because log_page_view_time holds one row per pageview whereas log_link_visit_action holds one row per hit. Query cost from EXPLAIN FORMAT=JSON is 617 k for the accurate path vs 735 k for legacy — comparable, with the row-count advantage offsetting the extra log_visit JOIN.

The archiver-side presence check (hasAccuratePageViewTimeRowsInPeriod) is covered by index_idsite_server_time (using_index=true, query cost 1.41) — it's essentially free per archive-day-per-site.

Storage cost

log_page_view_time averages ~178 B/row on disk including indices. Projected footprint: ~1.8 GB per 10 M pageviews. Row-purger integration is automatic: PageViewTime::hasIdVisitorColumn() = true + getDateTimeColumn() = 'server_time', so existing RawLogDao sweeps pick it up alongside log_link_visit_action.

Comparison to the older PoC (dev-18703-write-time-spent)

The earlier PoC took a different, simpler shape — worth comparing so the tradeoffs are explicit:

Concern	dev-18703	this PR (DEV-20451)
Row key	(idvisit, idsite, servertime)	(idvisit, idpageview) with UNIQUE index
Multi-tab attribution	Not supported — visit-level clock only	Supported — one row per tab via pv_id
Statements per hit	Two (SELECT MAX(servertime) INTO @var; UPDATE ... WHERE servertime=@var) plus INSERT IGNORE on new PV	One (UPDATE or INSERT ... ON DUPLICATE KEY UPDATE)
Session variables	Uses @max_servertime — needed the follow-up Aurora fix	None
Concurrency safety	SELECT then UPDATE window; another connection can insert a new row between the two statements	Atomic single-statement upsert on unique key + GREATEST() so out-of-order pings only grow
Cap on stalled tabs	None — time_spent = now - servertime, unbounded	LEAST(visit_standard_length, ...) applied inline
Row lookup on write	WHERE idvisit=? AND idsite=? — depends on the MAX(servertime) scan	Direct unique-key hit
Table row size	Wider (includes servertime used as effective key)	Narrower (~178 B including indices)
Schema migration	Not shipped — hand-created table only	core/Updates/5.13.0-b1.php + fresh-install schema

Concurrency prediction under real tracking load — the two most concurrency-sensitive scenarios are:

• Same tab, rapid hits (heartbeat + event + heartbeat within milliseconds): dev-18703 runs SELECT MAX(servertime) twice under lock; one connection may see stale data before the other's UPDATE lands. This PR uses a single UPDATE ... WHERE idvisit=? AND idpageview=? — InnoDB serializes the two on the same row lock, GREATEST guarantees the value only grows, no lost update.
• Two tabs, same visit, hits arriving simultaneously: dev-18703 uses a single visit-level row keyed on (idvisit, servertime) — the row is contested and one tab's time may overwrite the other. This PR keys per tab ((idvisit, idpageview)) so the two writes hit distinct rows. Zero contention between tabs.

Deadlock risk: none observed in local writer tests (concurrent multi-tab scenarios covered by the writer test suite pass consistently). Because every writer path targets exactly one row via a unique index, and no writer path takes locks on more than one row, the classic two-row cross-lock pattern that produces deadlocks cannot arise. The archiver reads with READ COMMITTED semantics via LogAggregator, so it doesn't take any row locks the writer would wait on.

What this PR still doesn't answer for performance

• Real production traffic was not replayed against this branch. The perf numbers above are from a single-node ddev MySQL 5.7 with a synthetic ~4 M row seed; a proper replay against real tracker traffic (queue-backed or otherwise) is worthwhile before shipping this as the on-by-default behaviour.
• The pv_time tracker parameter (deferred to a follow-up ticket) will add one more integer per tracker request. It is not measured here.

Why screenshots changed

CI produced 14 failed UI screenshots. Inspected all of them via the processed artifacts. They fall into three groups, each expected:

Group A — Time-on-page numbers moved (12 screenshots)
Bandwidth_actions_page_urls, Bandwidth_actions_page_urls_flat, Bandwidth_actions_page_titles, Live_visitor_profile_action_tooltip, Overlay_loaded, Overlay_page_link_clicks, Overlay_page_new_links, Overlay_page_change, Overlay_period_change, Overlay_row_evolution, Overlay_transitions, Overlay_loaded_from_actions.

Direct consequence of the metric change. The Avg. time on page column in the Pages report and the "Time on page" field in the Overlay sidebar / visitor-profile tooltip now come from sum_time_spent / nb_hits_with_time_spent instead of sum_time_spent / nb_hits, so the displayed durations shift. E.g. Bandwidth Pages report row test: 00:00:24 → 00:02:07; Overlay sidebar for page-1: 36s Avg. time on page where the legacy render showed a smaller value. Nothing else on these pages changed.

Group B — New config key in the rendered config file (1 screenshot)
UIIntegrationTest_admin_diagnostics_configfile — the System → Diagnostics config-file view dumps config/global.ini.php verbatim, so the new [Tracker] record_accurate_page_view_time = 1 line now appears in the screenshot.

Group C — New table in the schema (1 screenshot)
Installation_db_existing — the Installation "Some tables already exist" screen enumerates every matomo_* table found in the target DB. matomo_log_page_view_time now appears in that list because the migration creates it.

All three groups are the intended output of the PR; no unintended regressions were spotted in the pixels. Expected screenshots need to be resynced — the Bandwidth ones live in the plugin-Bandwidth submodule; Overlay / Live / Installation / UIIntegrationTest are in-tree.

Why some TimezonesTest fixtures dropped <idVisit>

Six tests/PHPUnit/System/expected/test_TimezonesTest_*__Live.getLastVisitsDetails_*.xml fixtures lost every <idVisit>N</idVisit> element in this PR's sync commit. This is not a regression — nothing on this branch changed how idVisit is emitted.

tests/PHPUnit/System/TimezonesTest.php explicitly declares:

'xmlFieldsToRemove' => ['idVisit'],

on each of the six Live.getLastVisitsDetails testcases. The Matomo test framework strips <idVisit> from the API response before comparing against the expected fixture, precisely because visit-ID numbering is flaky under the timezone permutations this test runs.

The old checked-in fixtures still had <idVisit>67</idVisit> etc. populated — the xmlFieldsToRemove rule was added to TimezonesTest after those files were last regenerated. CI on this branch ran the test, applied the removal rule, and produced a "clean" processed file. development:sync-system-test-processed copied that clean file over the stale one, so the diff looks like a data loss but is actually a long-overdue catch-up. The test framework has always been ignoring these values at assertion time, so no runtime behaviour changes.

Checklist

• [✔] I have understood, reviewed, and tested all AI outputs before use
• [✔] All AI instructions respect security, IP, and privacy rules