Scam-Qwen3-VL-RsLoRA

A Rank-Stabilized LoRA (rsLoRA) adapter for Qwen/Qwen3-VL-4B-Instruct, fine-tuned to detect crypto / financial scams in short-form social video (YouTube + TikTok), with a focus on Filipino / Philippine scam content. Given a video's frames plus its title and description, the model emits a chain-of-thought <think> rationale grounded in a policy criteria scheme (C1–C7) and a structured JSON verdict.

Model Details

Model Description

This is a PEFT/LoRA adapter only — it must be loaded on top of the frozen base model Qwen/Qwen3-VL-4B-Instruct. The base weights are never modified; only the rsLoRA adapter (~120 MB) is trained and distributed here.

The model takes a multimodal prompt (sampled video frames + text metadata) and produces:

1. A <think>...</think> block containing the evidence it found and which policy criteria (C1–C7) were hit, then
2. A JSON object: {"verdict": "Yes"|"No", "confidence": <float>, "category": <string>}.

• Developed by: Jules Gregory R. Agustin (HF: TamAko783)
• Model type: Vision-Language (multimodal) classifier / reasoner — rsLoRA adapter
• Task: Scam vs. not-scam detection over social-media video with rationale generation
• Language(s): English, Filipino/Tagalog (code-switched)
• License: CC0-1.0 (matches the training dataset)
• Finetuned from: Qwen/Qwen3-VL-4B-Instruct
• Adapter type: Rank-Stabilized LoRA (use_rslora=true), PEFT 0.19.1

Model Sources

• Repository: https://huggingface.co/TamAko783/Scam-Qwen3-VL-RsLoRA
• Base model: https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct
• Training dataset: OptiScam: Multimodal Filipino Scam Video Dataset

Uses

Direct Use

Screening / triage of short-form crypto and financial-promotion videos to flag likely scams (giveaways, airdrop bait, fake mining/doublers, get-rich-quick lures, gift-card and play-to-earn schemes), together with a human-readable rationale and policy-criteria citations.

Downstream Use

• A first-pass filter in a moderation pipeline, with the <think> trace surfaced to a human reviewer.
• A research baseline for multimodal scam detection and LLM-as-a-Judge reasoning evaluation.

Out-of-Scope Use

• Not an automated enforcement / takedown system. Use human-in-the-loop review.
• Not validated outside the crypto/financial scam domain or on long-form video.
• Not a financial-advice or investment-recommendation tool.
• Performance on languages or scam typologies under-represented in the training data is unverified.

Bias, Risks, and Limitations

• Over-flagging (false positives): the model sometimes labels legitimate educational/disclaimed content as a scam (e.g., real airdrop explainers, learn-to-earn guides, gift-card demos).
• Under-reading subtle signals (false negatives): misses domain spoofs (e.g. m1finance.8bxp97.net), P2P payment name mismatches, and Telegram/off-platform funnels that piggyback on legitimate branding.
• Occasional hallucination: rarely invents a domain name not present in the input (e.g. zoro-loot.com).
• Decoding artifacts: in a few cases the <think> block describes a textbook scam but the final verdict JSON says "No" (self-contradiction).
• Domain/locale bias: trained on a Philippine-focused corpus; cues may not transfer to other regions.

Recommendations

Always keep a human reviewer in the loop, treat the verdict as advisory, and read the <think> rationale to catch the failure modes above (over-flagging, contradictory verdicts, fabricated entities).

How to Get Started with the Model

This is an adapter, so you load the base model first, then attach the adapter.

pip install -U transformers peft accelerate torchvision qwen-vl-utils

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

BASE  = "Qwen/Qwen3-VL-4B-Instruct"
ADAPTER = "TamAko783/Scam-Qwen3-VL-RsLoRA"

# 1) Load the frozen base model
model = AutoModelForImageTextToText.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto",
)

# 2) Attach the rsLoRA adapter
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

# (optional) fuse for faster inference: model = model.merge_and_unload()

processor = AutoProcessor.from_pretrained(BASE)

# 3) Build a multimodal prompt: sampled video frames + metadata
SYSTEM = ("You are an expert scam-detection analyst. Use your native OCR ability "
          "to analyze the frames provided. Decide if the video is a scam based on "
          "YouTube's Scams policy and cite policy criteria (C1-C7).")

messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM}]},
    {"role": "user", "content": [
        # pass sampled frames (or a {"type": "video", "video": "<path>"} entry)
        {"type": "image", "image": "frame_00.jpg"},
        {"type": "image", "image": "frame_30.jpg"},
        {"type": "text", "text":
            "Title: <video title>\nDescription: <video description>\n\n"
            "Is this a scam video? Provide a rationale grounded in Policy Criteria (C1-C7)."},
    ]},
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Expected output shape:

<think>
Evidence: ...
Criteria hits (C1-C7): C2 - promises unbounded free crypto; C4 - fast-money lure ...
</think>
{"verdict": "Yes", "confidence": 0.95, "category": "Crypto Investment"}

Tips: the model was trained on up to 60 frames per video at max_pixels ≈ 200704. Thinking (<think>) mode benefits from max_new_tokens=1024. Greedy decoding (do_sample=False) is recommended for reproducible verdicts.

Training Details

Training Data

• Dataset: julesgregoryagustin/training-filipino-scam-dataset (CC0-1.0), ShareGPT multimodal format.
• ~2,000 YouTube/TikTok videos sampled at 60 frames each; ~1,600 train / held-out test (n=202 eval items).
• Each example: system prompt + video frames + title/description, with a gold yes/no label and a human-authored rationale ("Why Verdict" + "Criteria Hits", criteria C1–C7).

Training Procedure

Training Hyperparameters

Hyperparameter	Value
Adapter	Rank-Stabilized LoRA (use_rslora=true)
LoRA rank r	32
LoRA alpha	32
LoRA dropout	0.1
Effective scaling (α/√r)	≈ 5.657
Init	Gaussian
Learning rate	7e-5
Epochs	4
Max frames	60
Max pixels / frame	200704 (~448×448)
Precision	bf16 mixed precision
Base model	frozen
Final eval loss	0.884 (epoch 4)

Target modules: text attention (q_proj, k_proj, v_proj, o_proj), vision attention (qkv, attn.proj), and the multimodal projector / deepstack mergers (linear_fc1, linear_fc2).

Compute

Single NVIDIA A100 GPU (Google Colab), PEFT 0.19.1.

Evaluation

All metrics are on the held-out test split (n = 202).

Verdict accuracy

Configuration	Accuracy	Precision	Recall	F1	TP/FP/TN/FN
rsLoRA — full multimodal, thinking on (max_new_tokens=1024)	0.8119	0.8387	0.7723	0.8041	78/15/86/23
rsLoRA — greedy, no-thinking decode	0.7772	0.8182	0.7129	0.7619	72/16/85/29
Base Qwen3-VL-4B — zero-shot (no adapter)	0.7277	0.7556	0.6733	0.7120	68/22/79/33

The thinking-enabled rsLoRA adapter improves F1 by ~9 points over the zero-shot base model (0.804 vs 0.712) and tightens precision (fewer false positives: 15 vs 22).

Input-modality ablation (greedy, 256 tokens)

Inputs	Accuracy	Precision	Recall	F1	TP/FP/TN/FN
Frames only	0.7574	0.7364	0.8020	0.7678	81/29/72/20
Transcript + Title + Description	0.7228	0.7143	0.7426	0.7282	75/30/71/26
Transcript only	0.5446	0.5413	0.5842	0.5619	59/50/51/42

Takeaway: the visual signal dominates. Frames alone reach F1 ≈ 0.77, while transcript-only collapses toward chance (0.56) — scam cues (on-screen URLs, overlays, wallet QR codes) live in the pixels, which is exactly what the rsLoRA vision-tower adaptation targets.

Reasoning quality — LLM-as-a-Judge (n = 40 stratified)

Reasoning traces were audited with Claude Opus 4.7 as a cross-family judge under a reference-grounded binary rubric, treating reasoning quality as an inter-rater agreement problem against an outcome-derived rater.

Metric	Value
Percentage agreement (p_o)	0.9750 (39/40)
Cohen's κ	0.9500 — almost perfect (Landis–Koch)
BERTScore F1 (mean)	0.8696
ROUGE-L F1 (mean)	0.2580

Triangulation with reference-free similarity metrics shows a monotone TN > TP > FP > FN gradient (BERTScore F1: 0.9005 > 0.8824 > 0.8531 > 0.8425), and judge-Correct items score 0.893 vs 0.849 for judge-Incorrect — independent evidence that the model's reasoning is genuinely paraphrase-aligned with the human rationale when its verdict is right (not post-hoc rationalization).

Observed failure modes

Failure	Severity	Example
Over-flagging legitimate content (false positives)	high	misreads disclaimed airdrop/learn-to-earn videos as scams
Under-reading subtle scam signals (false negatives)	high	misses domain spoofs, P2P name mismatch, Telegram funnels
Fabricated entities (hallucination)	high	invented zoro-loot.com
Self-contradictory verdict (decoding bug)	medium	<think> says "scam" but JSON verdict = "No"

Policy criteria (C1–C7)

The rationales cite a YouTube-Scams-policy-derived criteria scheme. Codes observed in the model's output include: C2 (promises unbounded/free crypto), C3 (airdrop-link bait / off-platform redirect), C4 (fast-money / guaranteed-return lure), C5 (solicits wallet address / seed phrase), C6 (get-rich-quick hook), C7 (brand/channel impersonation).

Full legend: C1 Criminal claim, C2 Unbounded giveaway, C3 Off-site redirect, C4 Fast-money lure, C5 Harmful link, C6 Get-rich-quick, C7 Impersonation. Extended Filipino archetypes: P2E (Play-to-Earn), Task (Telegram / Group Task), and E-Wallet (GCash / Maya phishing). The criteria scheme is adapted from Kulsum et al. (arXiv:2509.23418).

Technical Specifications

Model Architecture and Objective

Causal-LM (CAUSAL_LM) rsLoRA adapter over the Qwen3-VL-4B-Instruct vision-language backbone. Objective: supervised fine-tuning to produce a <think> rationale + JSON scam verdict from video frames and text metadata. Processor: Qwen3VLProcessor (fps=2, patch size 16, merge size 2).

Framework versions

• PEFT 0.19.1
• Transformers (Qwen3-VL support)
• base_model: Qwen/Qwen3-VL-4B-Instruct

Citation

If you use this adapter, please cite the base model and dataset:

@misc{agustin2026scamqwen3vl,
  title  = {Scam-Qwen3-VL-RsLoRA: A Rank-Stabilized LoRA Adapter for Multimodal Scam Detection},
  author = {Agustin, Jules Gregory R.},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/TamAko783/Scam-Qwen3-VL-RsLoRA}}
}

Dataset and criteria inspiration:

Ummay Kulsum, Aafaq Sabir, Abhinaya S.B., and Anupam Das. "Beyond Metadata: Multimodal, Policy-Aware Detection of YouTube Scam Videos." ICWSM 2026. arXiv:2509.23418.

@inproceedings{kulsum2026beyond,
  title     = {Beyond Metadata: Multimodal, Policy-Aware Detection of YouTube Scam Videos},
  author    = {Kulsum, Ummay and Sabir, Aafaq and Abhinaya, S.B. and Das, Anupam},
  booktitle = {Proceedings of the International AAAI Conference on Web and Social Media (ICWSM)},
  year      = {2026},
  note      = {arXiv:2509.23418}
}

Model Card Authors

Jules Gregory R. Agustin (TamAko783)