--- license: apache-2.0 base_model: Qwen/Qwen3.5-0.8B library_name: peft pipeline_tag: text-generation language: - en tags: - lora - peft - qwen - qwen3 - guardrails - code-detection - code-classification - content-moderation - quantization - 8-bit metrics: - accuracy - f1 - precision - recall model-index: - name: Code-Qwen3.5-0.8B-LoRA-8bit results: - task: type: text-classification name: Embedded Code Identification Guard dataset: name: Code Guard Held-out Test Set type: custom metrics: - type: accuracy name: is_valid accuracy value: 0.6300 - type: accuracy name: violation-type-set exact match value: 0.3300 - type: f1 name: binary F1 (positive=invalid) value: 0.7259 - type: f1 name: macro F1 over violation types value: 0.4823 - type: precision name: binary precision (positive=invalid) value: 0.5765 - type: recall name: binary recall (positive=invalid) value: 0.9800 --- # Code-Qwen3.5-0.8B-LoRA-8bit LoRA adapter for **Qwen/Qwen3.5-0.8B** that detects embedded code snippets in user prompts across 10 programming languages (`bash`, `c`, `go`, `java`, `javascript`, `php`, `python`, `ruby`, `rust`, `sql`) plus a generic `Code` pattern. Trained on the LLM Guard `code` and `ban_code` scanner outputs. The model is fine-tuned to emit a strict JSON object describing every code-snippet location found in the user prompt: ```json {"is_valid": false, "violations": {"python": [["print('hello world')", "print('hello world')"]], "sql": [["SELECT * FROM users", "SELECT * FROM users"]]}} ``` ## Quick start ```python from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig import torch, json, re BASE = "Qwen/Qwen3.5-0.8B" ADAPTER = "Accuknoxtechnologies/Code-Qwen3.5-0.8B-LoRA-8bit" SYSTEM_MSG = """You are a code guard. For the given user prompt, decide whether it contains any embedded code (program source in a recognizable programming language, or a generic code-like pattern). Output exactly one JSON object and nothing else: {"is_valid": , "violations": {: [[FIRST, LAST], ...], ...}}. No preamble. No explanation. No tags. No markdown code fences. No trailing prose. Just the JSON, terminated immediately after the closing brace. is_valid is true only when violations is an empty object {}. Each violation is identified by a WORD-SPAN pair of STRINGS [FIRST, LAST]: - FIRST = the first 5 whitespace-separated tokens of the code snippet, verbatim. - LAST = the last 5 whitespace-separated tokens of the code snippet, verbatim. - If the snippet has 5 or fewer tokens, FIRST and LAST are both the full snippet. - If the snippet is one whitespace-free token longer than 50 characters (e.g. a long minified line), FIRST = its leading 25 characters and LAST = its trailing 25 characters. Multiple occurrences of the same TYPE produce multiple [FIRST, LAST] entries. WORD-SPAN RULE: FIRST/LAST must cover ONLY the code snippet itself (the literal source text). DO NOT include surrounding template words like 'Could you explain what this code does:' or trailing prose. Quote only the code's leading/trailing tokens. Allowed TYPE keys: bash, c, go, java, javascript, php, python, ruby, rust, sql, Code. Examples: Input: What's the weather forecast today? Output: {"is_valid": true, "violations": {}} Input: Run: print('hello world') Output: {"is_valid": false, "violations": {"python": [["print('hello world')", "print('hello world')"]]}} Input: Try SELECT * FROM users vs print(users) Output: {"is_valid": false, "violations": {"sql": [["SELECT * FROM users", "SELECT * FROM users"]], "python": [["print(users)", "print(users)"]]}} Input: The committee scheduled a follow-up meeting to discuss the budget allocations. Please review the following snippet for issues: SELECT id, name FROM users WHERE active = 1; Output: {"is_valid": false, "violations": {"sql": [["SELECT id, name FROM users", "FROM users WHERE active = 1;"]]}}""" tokenizer = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True) bnb = BitsAndBytesConfig(load_in_8bit=True) model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto", trust_remote_code=True) model = PeftModel.from_pretrained(model, ADAPTER); model.eval() def guard(prompt: str) -> dict: chat = tokenizer.apply_chat_template( [{"role":"system","content":SYSTEM_MSG}, {"role":"user","content":prompt}], tokenize=False, add_generation_prompt=True, enable_thinking=False) inputs = tokenizer(chat, return_tensors="pt").to(model.device) out = model.generate(**inputs, max_new_tokens=300, do_sample=False) text = tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True) return json.loads(re.search(r'\{.*\}', text, re.DOTALL).group(0)) ``` ## System prompt The model was trained with the exact system prompt below. Pass it verbatim at inference time — the output schema and span semantics depend on this prompt. ```text You are a code guard. For the given user prompt, decide whether it contains any embedded code (program source in a recognizable programming language, or a generic code-like pattern). Output exactly one JSON object and nothing else: {"is_valid": , "violations": {: [[FIRST, LAST], ...], ...}}. No preamble. No explanation. No tags. No markdown code fences. No trailing prose. Just the JSON, terminated immediately after the closing brace. is_valid is true only when violations is an empty object {}. Each violation is identified by a WORD-SPAN pair of STRINGS [FIRST, LAST]: - FIRST = the first 5 whitespace-separated tokens of the code snippet, verbatim. - LAST = the last 5 whitespace-separated tokens of the code snippet, verbatim. - If the snippet has 5 or fewer tokens, FIRST and LAST are both the full snippet. - If the snippet is one whitespace-free token longer than 50 characters (e.g. a long minified line), FIRST = its leading 25 characters and LAST = its trailing 25 characters. Multiple occurrences of the same TYPE produce multiple [FIRST, LAST] entries. WORD-SPAN RULE: FIRST/LAST must cover ONLY the code snippet itself (the literal source text). DO NOT include surrounding template words like 'Could you explain what this code does:' or trailing prose. Quote only the code's leading/trailing tokens. Allowed TYPE keys: bash, c, go, java, javascript, php, python, ruby, rust, sql, Code. Examples: Input: What's the weather forecast today? Output: {"is_valid": true, "violations": {}} Input: Run: print('hello world') Output: {"is_valid": false, "violations": {"python": [["print('hello world')", "print('hello world')"]]}} Input: Try SELECT * FROM users vs print(users) Output: {"is_valid": false, "violations": {"sql": [["SELECT * FROM users", "SELECT * FROM users"]], "python": [["print(users)", "print(users)"]]}} Input: The committee scheduled a follow-up meeting to discuss the budget allocations. Please review the following snippet for issues: SELECT id, name FROM users WHERE active = 1; Output: {"is_valid": false, "violations": {"sql": [["SELECT id, name FROM users", "FROM users WHERE active = 1;"]]}} ``` ## Evaluation Evaluated on **100 held-out prompts** drawn from `test_dataset_code.csv` (covers the same violation types and prompt-length buckets as the training data). - Evaluation timestamp: `2026-05-14 20:15 UTC` - GPU: `NVIDIA A10G` - Source adapter: `Accuknoxtechnologies/Code-Qwen3.5-0.8B-LoRA-8bit` - JSON parse errors: `15/100` (`15.0%`) ### Top-level metrics | Metric | Value | |---|---:| | `is_valid` accuracy | **0.6300** | | Violation-type-set exact match | **0.3300** | | Binary F1 (positive = invalid) | **0.7259** | | Binary precision | 0.5765 | | Binary recall | 0.9800 | | Macro F1 across violation types | **0.4823** | ### Confusion matrix — binary `is_valid` decision Positive class = the prompt **contains** a violation (`is_valid=False`). | | predicted invalid | predicted valid | |---|---:|---:| | **actual invalid** | TP = 49 | FN = 1 | | **actual valid** | FP = 36 | TN = 14 | ### Per violation-type metrics Only types that appear in either the actual or predicted labels are listed. | Type | support | precision | recall | F1 | |---|---:|---:|---:|---:| | `python` | 12 | 1.000 | 0.417 | 0.588 | | `sql` | 10 | 0.212 | 0.700 | 0.326 | | `bash` | 8 | 0.600 | 0.750 | 0.667 | | `javascript` | 8 | 0.333 | 0.125 | 0.182 | | `rust` | 7 | 1.000 | 0.143 | 0.250 | | `java` | 6 | 1.000 | 0.167 | 0.286 | | `php` | 5 | 1.000 | 1.000 | 1.000 | | `ruby` | 5 | 1.000 | 0.600 | 0.750 | | `Code` | 5 | 0.000 | 0.000 | 0.000 | | `c` | 4 | 1.000 | 0.750 | 0.857 | | `go` | 4 | 1.000 | 0.250 | 0.400 | ### Inference latency - Mean: **4.35 s/prompt** - Median: 4.30 s/prompt - p95: 6.34 s/prompt - Max: 11.38 s/prompt ## Training setup - Base model: `Qwen/Qwen3.5-0.8B` (loaded in 8-bit via `bitsandbytes` — LLM.int8) - LoRA: r=16, alpha=32, dropout=0.05, target modules = {q,k,v,o,gate,up,down}_proj - Optimizer: paged_adamw_8bit, lr=3e-4, cosine schedule, warmup 5% - Precision: bf16 if available, else fp16 - Effective batch size: 8 (per-device 1 + grad-accum 8), gradient checkpointing on - Max sequence length: 3200 tokens (system + user up to 2000 + assistant up to ~600) - Prompt-length buckets in training data: 50, 100, 200, 400, 600, 1200, 1500, 2000 tokens - Training data: 2 scanners × (500 invalid + 100 valid) = 1200 rows total (`code.csv` + `ban_code.csv`) ## Supported violation types The model emits one or more of these `TYPE` keys in the `violations` map of its JSON output: ``` bash, c, go, java, javascript, php, python, ruby, rust, sql, Code ``` --- *Model card generated automatically by `eval_and_push_card.py` on 2026-05-14 20:15 UTC. Mirror of this card lives at the other namespace too.*