Aureliolo/game-review-reader

Game Review Reader

Classifies one claim from a Steam review: which of 26 subjects it is about, and whether it is praise, a complaint or neutral. Fine-tuned from intfloat/multilingual-e5-large-instruct, run e5inst-pool-person-rules-s1. Part of SteamGauge.

Use

• model.onnx takes input_ids and attention_mask for a pair tokenised with tokenizer.json: the claim, then the text of its review around it, at most 128 tokens. For a game played only in a VR headset, the review text starts with "Played in a VR headset."
• It returns subject_logits (26, in the order of subjects in reader.json), polarity_logits (praise, complaint, neutral) and pooled.
• reader.json holds the confidence each subject (thresholds) and each language (language_thresholds) needs before the reader answers. A claim gets an answer when its top subject's probability clears both thresholds. A language with no threshold, or missing from the list, gets no answers.

Subjects: accessibility, atmosphere, audio, bugs, community, compatibility, content, controls, difficulty, gameplay, genre, graphics, language, licensing, mods, monetisation, multiplayer, offtopic, performance, policy, price, story, tutorial, updates, verdict, vr.

Results

On 5,080 claims from games held out of training:

• Subject accuracy 0.781, macro F1 0.710.
• Polarity macro F1 0.855.
• Calibration error 0.074; area under the risk-coverage curve 0.074.
• With its thresholds, out of fold over 40,327 claims of 59 games: answers 89% of claims, accuracy 0.793, macro F1 0.787.
• Lowest per-subject F1: accessibility 0.33, vr 0.40, community 0.44, genre 0.60, licensing 0.64.
• No answers in: bulgarian, danish, dutch, finnish, greek, indonesian, norwegian, portuguese, romanian, vietnamese.

Sizes

Reader	Parameters	Download	Accuracy at 80% coverage	Accuracy at 90% coverage	GPU memory	One large game on the GPU	One small game on the CPU
small	118M	244 MB	77.9%	73.8%	1.2 GB	62 s	36 s
standard	559M	1.1 GB	84.7%	82.0%	2.6 GB	125 s	285 s

• Accuracy at a coverage: 458 claims from ten games held out of training, labelled under the current category sheet. Each reader answers the given share of claims it is most confident about, scored on DirectML against reference/frontier-key.json (frontier.py reader).
• GPU memory and one large game on the GPU: one read of game 920210 (117,664 claims) through the desktop app on DirectML, on an otherwise idle NVIDIA GeForce RTX 4090. Memory is the peak during the read minus what the GPU held before.
• One small game on the CPU: 1,416 claims of game 1888930 on the CPU only, loading included, on an AMD Ryzen 9 5950X with other work running.

Training

36,017 labelled claims, with 4,310 more for validation, from Aureliolo/game-review-claims, and a larger reader's answers on 246,989 unlabelled claims from the same games' reviews, which are not published. SteamGauge commit 409ca2b52b64.

Limits

The training labels were written by Claude models from a category sheet, and the results above measure agreement with those labels.

Licence

Apache-2.0. Base model intfloat/multilingual-e5-large-instruct: MIT.