Stage Miril-DroneVLM-2B-2 release

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/qualitative_examples.png filter=lfs diff=lfs merge=lfs -text
+technical_report.pdf filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,201 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction,
+and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by
+the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all
+other entities that control, are controlled by, or are under common
+control with that entity. For the purposes of this definition,
+"control" means (i) the power, direct or indirect, to cause the
+direction or management of such entity, whether by contract or
+otherwise, or (ii) ownership of fifty percent (50%) or more of the
+outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity
+exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications,
+including but not limited to software source code, documentation
+source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical
+transformation or translation of a Source form, including but
+not limited to compiled object code, generated documentation,
+and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or
+Object form, made available under the License, as indicated by a
+copyright notice that is included in or attached to the work
+(an example is provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object
+form, that is based on (or derived from) the Work and for which the
+editorial revisions, annotations, elaborations, or other modifications
+represent, as a whole, an original work of authorship. For the purposes
+of this License, Derivative Works shall not include works that remain
+separable from, or merely link (or bind by name) to the interfaces of,
+the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including
+the original version of the Work and any modifications or additions
+to that Work or Derivative Works thereof, that is intentionally
+submitted to Licensor for inclusion in the Work by the copyright owner
+or by an individual or Legal Entity authorized to submit on behalf of
+the copyright owner. For the purposes of this definition, "submitted"
+means any form of electronic, verbal, or written communication sent
+to the Licensor or its representatives, including but not limited to
+communication on electronic mailing lists, source code control systems,
+and issue tracking systems that are managed by, or on behalf of, the
+Licensor for the purpose of discussing and improving the Work, but
+excluding communication that is conspicuously marked or otherwise
+designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity
+on behalf of whom a Contribution has been received by Licensor and
+subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+copyright license to reproduce, prepare Derivative Works of,
+publicly display, publicly perform, sublicense, and distribute the
+Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+(except as stated in this section) patent license to make, have made,
+use, offer to sell, sell, import, and otherwise transfer the Work,
+where such license applies only to those patent claims licensable
+by such Contributor that are necessarily infringed by their
+Contribution(s) alone or by combination of their Contribution(s)
+with the Work to which such Contribution(s) was submitted. If You
+institute patent litigation against any entity (including a
+cross-claim or counterclaim in a lawsuit) alleging that the Work
+or a Contribution incorporated within the Work constitutes direct
+or contributory patent infringement, then any patent licenses
+granted to You under this License for that Work shall terminate
+as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+Work or Derivative Works thereof in any medium, with or without
+modifications, and in Source or Object form, provided that You
+meet the following conditions:
+
+(a) You must give any other recipients of the Work or
+Derivative Works a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices
+stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works
+that You distribute, all copyright, patent, trademark, and
+attribution notices from the Source form of the Work,
+excluding those notices that do not pertain to any part of
+the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its
+distribution, then any Derivative Works that You distribute must
+include a readable copy of the attribution notices contained
+within such NOTICE file, excluding those notices that do not
+pertain to any part of the Derivative Works, in at least one
+of the following places: within a NOTICE text file distributed
+as part of the Derivative Works; within the Source form or
+documentation, if provided along with the Derivative Works; or,
+within a display generated by the Derivative Works, if and
+wherever such third-party notices normally appear. The contents
+of the NOTICE file are for informational purposes only and
+do not modify the License. You may add Your own attribution
+notices within Derivative Works that You distribute, alongside
+or as an addendum to the NOTICE text from the Work, provided
+that such additional attribution notices cannot be construed
+as modifying the License.
+
+You may add Your own copyright statement to Your modifications and
+may provide additional or different license terms and conditions
+for use, reproduction, or distribution of Your modifications, or
+for any such Derivative Works as a whole, provided Your use,
+reproduction, and distribution of the Work otherwise complies with
+the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+any Contribution intentionally submitted for inclusion in the Work
+by You to the Licensor shall be under the terms and conditions of
+this License, without any additional terms or conditions.
+Notwithstanding the above, nothing herein shall supersede or modify
+the terms of any separate license agreement you may have executed
+with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+names, trademarks, service marks, or product names of the Licensor,
+except as required for reasonable and customary use in describing the
+origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+agreed to in writing, Licensor provides the Work (and each
+Contributor provides its Contributions) on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+implied, including, without limitation, any warranties or conditions
+of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+PARTICULAR PURPOSE. You are solely responsible for determining the
+appropriateness of using or redistributing the Work and assume any
+risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+whether in tort (including negligence), contract, or otherwise,
+unless required by applicable law (such as deliberate and grossly
+negligent acts) or agreed to in writing, shall any Contributor be
+liable to You for damages, including any direct, indirect, special,
+incidental, or consequential damages of any character arising as a
+result of this License or out of the use or inability to use the
+Work (including but not limited to damages for loss of goodwill,
+work stoppage, computer failure or malfunction, or any and all
+other commercial damages or losses), even if such Contributor
+has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+the Work or Derivative Works thereof, You may choose to offer,
+and charge a fee for, acceptance of support, warranty, indemnity,
+or other liability obligations and/or rights consistent with this
+License. However, in accepting such obligations, You may act only
+on Your own behalf and on Your sole responsibility, not on behalf
+of any other Contributor, and only if You agree to indemnify,
+defend, and hold each Contributor harmless for any liability
+incurred by, or claims asserted against, such Contributor by reason
+of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+To apply the Apache License to your work, attach the following
+boilerplate notice, with the fields enclosed by brackets "[]"
+replaced with your own identifying information. (Don't include
+the brackets!) The text should be enclosed in the appropriate
+comment syntax for the file format. We also recommend that a
+file or class name and description of purpose be included on the
+same "printed page" as the copyright notice for easier
+identification within third-party archives.
+
+Copyright [yyyy] [name of copyright owner]
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

NOTICE

@@ -0,0 +1,15 @@
+Miril-DroneVLM-2B-2
+Copyright 2026 Miril
+
+Miril-DroneVLM-2B-2 is an aerial vision-language model derived from
+google/gemma-4-E2B-it by Google DeepMind.
+
+The base model google/gemma-4-E2B-it is published by Google DeepMind
+under the Apache License, Version 2.0.
+
+Miril modifications adapt the base model for aerial scene understanding,
+drone visual question answering, typed JSON routing, and structured
+image-relative coordinate outputs.
+
+This distribution is not endorsed by or affiliated with Google,
+Google DeepMind, or the Gemma authors.

README.md

@@ -0,0 +1,571 @@
+---
+license: apache-2.0
+base_model: google/gemma-4-E2B-it
+library_name: transformers
+pipeline_tag: image-text-to-text
+language:
+  - en
+tags:
+  - aerial-vlm
+  - drone
+  - uav
+  - vision-language-model
+  - aerial-imagery
+  - scene-understanding
+  - visual-question-answering
+  - visual-grounding
+  - robotics
+  - edge-ai
+  - structured-output
+  - open-weights
+  - miril
+---
+
+# Miril-DroneVLM-2B-2
+
+### Drones can talk.
+
+**Enabling drones to reason and talk about their environment.**
+
+Ask a drone what it sees. Get an answer software can use.
+
+Miril-DroneVLM-2B-2 is an open-weight, 2B-class aerial vision-language model from [Miril.ai](https://miril.ai). Give it an overhead image and an ordinary English question. It returns one of four machine-readable JSON responses: a whole-scene caption, a factual answer, a landing or delivery location, or a point on one visible target.
+
+A drone camera can collect more imagery than a person can inspect frame by frame. This model explores a practical interface for that stream: ask what is present, what is happening, where an object is, or which part of the frame deserves review, then pass the typed result to an operator interface or separately validated application.
+
+The target applications are civilian: delivery, first response, infrastructure inspection, agriculture, construction, mapping, and disaster recovery. Miril-DroneVLM-2B-2 is an edge-oriented research model for testing how those systems can become easier to supervise and integrate. It is a perception component, not a flight controller.
+
+Try the image and video demo:
+
+**[Miril-DroneVLM-2B-2-Demo](https://huggingface.co/spaces/MirilAI/Miril-DroneVLM-2B-2-Demo)**
+
+## Four Questions, Four Usable Results
+
+![Caption, answer, pointing, and landing examples](./assets/qualitative_examples.png)
+
+These are deterministic release-checkpoint MLX 8-bit outputs on attributed Pexels images
+outside the training corpus. Each panel shows the ordinary user question, the
+parsed response, and any drawable model coordinate. The examples were manually
+screened for presentation and are not an accuracy sample; the complete
+held-out results appear below.
+
+## One Question In, One Typed Result Out
+
+The user does **not** choose a schema or append formatting instructions. The user asks a normal question:
+
+```text
+What am I looking at?
+```
+
+```text
+How many light vehicles are visible?
+```
+
+```text
+Choose a place to lower this parcel.
+```
+
+```text
+Track the white car.
+```
+
+The application supplies the fixed system contract from [`router_contract.py`](./router_contract.py). The model interprets the requested operation and returns exactly one bare JSON object.
+
+The public runtime contract is `typed_json_router_v2`. Pin that contract file with the model revision when integrating the model into another application.
+
+| Requested result | JSON `type` | Secondary key | Downstream use |
+| --- | --- | --- | --- |
+| General read of the whole image | `caption` | none | Display or index a scene description |
+| Fact, count, presence, appearance, hazard, or verbal explanation | `answer` | none | Display or consume a textual answer |
+| Choose one landing or delivery area | `location` | `intent: landing` or `delivery` | Review a selected point or broad direction |
+| Locate, mark, or track one target | `pointing` | `action: locate`, `point`, or `track` | Draw a point or initialize downstream tracking |
+
+This separation is the point of the release. The response identifies its own type, so downstream code can dispatch deterministically instead of trying to classify free-form prose after generation.
+
+## Prompting In Plain English
+
+Prompt for the result you need, not for the JSON structure:
+
+| What you want | Natural examples | Response |
+| --- | --- | --- |
+| A read of the complete frame | “What am I looking at?”, “Describe the whole scene.” | `caption` |
+| A fact or explanation | “Any buses?”, “How many people?”, “What should I avoid here?” | `answer` |
+| A candidate landing or delivery area | “Where could I land?”, “Pick a clear place for this parcel.” | `location` |
+| One visible target | “Where is the white car?”, “Mark the storage tank.”, “Track that truck.” | `pointing` |
+
+These are intent families, not a keyword list. For example, “Where are the cars concentrated?” asks for a verbal answer, while “Where is the white car?” asks for one point.
+
+### `caption`
+
+Use a general request for the whole frame:
+
+```text
+What am I looking at?
+```
+
+```json
+{
+  "type": "caption",
+  "caption": "From above, one built-up structure anchors the upper-left corner of a tidy property, bordered by asphalt, bright grass, driveways, and trimmed greenery."
+}
+```
+
+### `answer`
+
+Use a question whose result should be words rather than one image point. This includes counts, object presence, colors, scene type, relative position, risks, suitability, and areas to avoid.
+
+```text
+Give the visible tally for utility poles.
+```
+
+```json
+{
+  "type": "answer",
+  "answer": "Six utility poles can be seen around the scene."
+}
+```
+
+`Where` does not automatically mean pointing. “Where are the vehicles concentrated?” asks for a verbal description and routes to `answer`. “Where is the white car?” asks for one concrete image point and routes to `pointing`.
+
+### `location`
+
+Use this only when asking the model to choose an area for landing a drone or placing a delivery. A location response may contain a precise point, a broad directional cue, or no point at all. Its status tells the application which interpretation is valid.
+
+```text
+Find a usable parcel drop location, if the view contains one.
+```
+
+A broad directional result can look like this:
+
+```json
+{
+  "type": "location",
+  "intent": "delivery",
+  "caption": "Coarse grid direction only, not a recommended landing or delivery spot. General safest-looking region: the open bare area on the right side of the road, away from vegetation and the building.",
+  "coordinate_system": "gemma_relative_0_1000_yx",
+  "point_2d": [500, 750],
+  "point_semantics": "representative_point",
+  "pointing_mode": "coarse_grid_direction",
+  "status": "coarse_direction",
+  "x": 750,
+  "y": 500
+}
+```
+
+That output identifies a direction in the frame. It does **not** identify an approved parcel drop or landing spot.
+
+### `pointing`
+
+Use this when one concrete visible point is the requested result.
+
+```text
+Where in the image is the center of the white-roofed structure?
+```
+
+```json
+{
+  "type": "pointing",
+  "action": "locate",
+  "caption": "The white-roofed structure is centered in the lower-right.",
+  "coordinate_system": "gemma_relative_0_1000_yx",
+  "point_2d": [936, 956],
+  "point_semantics": "specific_point",
+  "pointing_mode": "precise_point",
+  "status": "target_found",
+  "x": 956,
+  "y": 936
+}
+```
+
+The action follows the user’s intent:
+
+| User intent | Typical wording | `action` |
+| --- | --- | --- |
+| Find or look at one target | “Where is…”, “Find…”, “Look at…” | `locate` |
+| Put a marker on one target | “Point at…”, “Mark…”, “Highlight…” | `point` |
+| Supply the first point for temporal tracking | “Track…”, “Follow…”, “Lock on…” | `track` |
+
+The model provides only the initial image point for `track`. Your application performs tracking across later video frames. The included live demo starts from a 60 x 60 region in the processed model frame, maps that region back to source-video pixels, and removes the marker if optical flow fails or any edge of the tracked box touches the frame.
+
+## Coordinate Contract
+
+Spatial responses use `gemma_relative_0_1000_yx`:
+
+- `x=0` is the left edge and `x=1000` is the right edge.
+- `y=0` is the top edge and `y=1000` is the bottom edge.
+- `point_2d` is `[y, x]`.
+- `x` and `y` repeat the same values for simpler downstream access.
+
+There are three visual outcomes. They are deliberately different because a broad cue, a grounded point, and an abstention must not look interchangeable in an operator interface.
+
+### Precise point
+
+`pointing_mode: precise_point` is drawable only when the response is schema-valid and uses `status: target_found`. It represents one image-grounded point. A UI may draw a compact reticle there. For a `track` request it may also initialize a tracker.
+
+### Coarse direction
+
+`pointing_mode: coarse_grid_direction` is allowed only for `location`. It uses a 3 x 3 grid where both coordinates are exactly `250`, `500`, or `750`.
+
+Treat it as “look in this part of the frame,” not “land here.” Draw a broad region cue, never a precision reticle. Do not initialize a tracker or send it to a controller as a target.
+
+### No point
+
+`pointing_mode: none`, invalid JSON, an invalid schema, or a non-target status means there is no drawable point. Coordinate fields must be null. If a malformed response combines a non-target status with coordinates, ignore those coordinates.
+
+Interpret `status` by response type:
+
+| Response | Status | Meaning |
+| --- | --- | --- |
+| `location` | `target_found` | One image-grounded landing or delivery candidate was selected |
+| `location` | `coarse_direction` | Broad 3 x 3 directional context only |
+| `location` | `no_safe_area` | No usable area should be marked |
+| `location` | `unknown` | The image does not support a reliable judgment |
+| `pointing` | `target_found` | One requested target was grounded |
+| `pointing` | `no_target` | The requested object is absent |
+| `pointing` | `no_matching_target` | Objects may be present, but none satisfy the requested relation |
+| `pointing` | `ambiguous_target` | More than one target fits and the request does not identify one |
+| `pointing` | `unknown` | The image does not support a reliable point |
+
+The included validator enforces these rules before returning a dispatch-safe payload. Applications should still retain the original model text for audit rather than silently repairing it.
+
+## Mixed Requests
+
+The fixed router applies this precedence:
+
+1. A request to choose a landing or delivery area routes to `location`.
+2. Otherwise, an explicit request for one concrete point routes to `pointing`.
+3. Otherwise, a factual or verbal question routes to `answer`.
+4. Otherwise, a whole-scene read routes to `caption`.
+
+If several indistinguishable targets are named, the model should preserve the requested pointing action but return `ambiguous_target` with null coordinates instead of choosing arbitrarily. If different pointing actions are mixed for one target, `track` takes priority over `point`, and `point` takes priority over `locate`.
+
+## Loader Compatibility
+
+Use `transformers>=5.12.1`. Gemma 4 E2B declares 35 language layers and 20 shared-KV layers. Layers 15 through 34 reuse key/value states, so a correct export does not contain separate `k_proj`, `v_proj`, or `k_norm` tensors for those shared layers. Release validation checks this layout directly and rejects both genuinely missing owner tensors and redundant shared-layer tensors.
+
+The retained [shared-KV export audit](./shared_kv_export_report.json) records
+the expected and observed tensor owners for this exact artifact.
+
+## Quickstart
+
+Download the small runtime helpers from this repository, install their dependencies, and ask a plain-English question:
+
+```bash
+hf download MirilAI/Miril-DroneVLM-2B-2 inference.py router_contract.py requirements.txt --local-dir miril-drone-runtime
+python -m pip install -r miril-drone-runtime/requirements.txt
+python miril-drone-runtime/inference.py \
+  --image drone_frame.jpg \
+  --prompt "Where is the white car?"
+```
+
+The script:
+
+- sends the exact fixed router prompt as a system message,
+- sends the image and ordinary English question as the user message,
+- decodes deterministically,
+- requires valid bare JSON,
+- validates the complete response schema,
+- refuses to dispatch or draw malformed coordinates.
+
+For direct integration, import `TYPED_JSON_ROUTER_SYSTEM_PROMPT`, `typed_response_errors`, and `drawable_point` from [`router_contract.py`](./router_contract.py).
+
+### Experimental spoken question
+
+The inherited Gemma audio path can be exercised with the same image and router contract. This checkpoint was fine-tuned with typed questions, so use the published typed-versus-spoken benchmark to judge the modality gap rather than assuming parity. Install `ffmpeg`, then run:
+
+```bash
+python miril-drone-runtime/inference.py \
+  --image drone_frame.jpg \
+  --audio spoken_question.mp3 \
+  --audio-transcript "Where is the white car?"
+```
+
+`--audio-transcript` is optional provenance for the returned audit record. It is never supplied to the model; the user turn contains the image and decoded audio waveform only.
+
+## Image And Video Overlays
+
+The same black-side-panel presentation used by the demo is included as runnable code. Labels are bold green, every response is shown beside the question that produced it, and the panel is half the image width.
+
+```bash
+hf download MirilAI/Miril-DroneVLM-2B-2 \
+  inference.py router_contract.py image_overlay.py video_overlay.py requirements.txt \
+  --local-dir miril-drone-runtime
+```
+
+Image:
+
+```bash
+python miril-drone-runtime/image_overlay.py \
+  --image drone_frame.jpg \
+  --output overlay.jpg \
+  --prompt "What am I looking at?" \
+  --prompt "Are any people or light vehicles visible?" \
+  --prompt "Where is the white car?"
+```
+
+Video, with one inference pass every five seconds:
+
+```bash
+python miril-drone-runtime/video_overlay.py \
+  --input drone_video.mp4 \
+  --output drone_video_overlay.mp4 \
+  --interval-seconds 5 \
+  --crop center-square \
+  --crop-size 1000 \
+  --prompt "What am I looking at?" \
+  --prompt "Track the white car."
+```
+
+The renderer preserves the source frame rate, reveals each answer over 12 frames without changing its font size during the reveal, tracks valid precise points between inference calls, hides failed or edge-adjacent tracks, and never tracks a coarse direction or non-target response.
+
+## Evaluation
+
+Release evaluation has three complementary scopes:
+
+1. **Complete held-out validation:** every V2 release artifact is evaluated on every accepted validation request. This is the primary quantization-regression check for merged BF16, CUDA bnb8, CUDA bnb4, MLX 8-bit, and MLX 4-bit.
+2. **Independent route benchmark:** a separately authored held-out benchmark balances captioning, factual answers, landing selection, delivery selection, object location, visual pointing, and tracker initialization. It supports controlled V1/V2 and cross-runtime comparison without reusing training questions.
+3. **Cleaned held-out audit:** every unchanged release artifact is evaluated after a stricter review removes pointing targets that are too small under the model's actual image-preprocessing contract. Validation and test are complete, and strict-cleaned and coverage-matched views are reported separately.
+
+All scopes check route selection, strict JSON, schema validity, text overlap, status and mode agreement, null-coordinate discipline, coarse-grid agreement, and precise-coordinate recovery. No single aggregate can show all of those behaviors, so the release reports them separately. Publication requires complete prediction coverage and the exact case-file hash for every reported artifact.
+
+<!-- MIRIL_RELEASE_METRICS_START -->
+![Generation comparison](./assets/generation_comparison.svg)
+
+### Generation comparison
+
+| Metric | V1 merged BF16 | Merged BF16 |
+|---|---:|---:|
+| Valid JSON | 92.0% | 100.0% |
+| Schema valid | 21.8% | 96.1% |
+| Route accuracy | 21.8% | 94.8% |
+| Caption / answer F1 | 26.2% | 38.2% |
+| Spatial status | 2.9% | 79.2% |
+| Precise target retained | 7.5% | 50.2% |
+| Point within 100 | 3.9% | 33.4% |
+| Coarse direction exact | 0.0% | 35.0% |
+| No-target discipline | 43.9% | 93.4% |
+
+![Deployment comparison](./assets/deployment_comparison.svg)
+
+### Deployment comparison
+
+| Metric | Merged BF16 | CUDA bnb8 | CUDA bnb4 | MLX 8-bit | MLX 4-bit |
+|---|---:|---:|---:|---:|---:|
+| Valid JSON | 100.0% | 99.1% | 96.7% | 100.0% | 100.0% |
+| Schema valid | 96.1% | 93.6% | 92.1% | 94.6% | 55.8% |
+| Route accuracy | 94.8% | 91.6% | 91.2% | 93.4% | 56.9% |
+| Caption / answer F1 | 38.2% | 19.6% | 37.2% | 37.3% | 35.1% |
+| Spatial status | 79.2% | 63.4% | 69.7% | 77.9% | 77.3% |
+| Precise target retained | 50.2% | 42.4% | 53.1% | 51.2% | 46.9% |
+| Point within 100 | 33.4% | 8.0% | 29.7% | 31.0% | 25.1% |
+| Coarse direction exact | 35.0% | 9.3% | 30.2% | 32.1% | 27.6% |
+| No-target discipline | 93.4% | 89.6% | 87.5% | 92.6% | 91.6% |
+
+### Complete held-out validation
+
+![Complete validation comparison](./assets/full_validation_comparison.svg)
+
+| Metric | Merged BF16 | CUDA bnb8 | CUDA bnb4 | MLX 8-bit | MLX 4-bit |
+|---|---:|---:|---:|---:|---:|
+| Valid JSON | 99.9% | 99.7% | 99.0% | 100.0% | 100.0% |
+| Schema valid | 99.9% | 99.7% | 98.7% | 100.0% | 53.0% |
+| Route accuracy | 99.9% | 99.6% | 98.6% | 100.0% | 53.2% |
+| Caption / answer F1 | 38.6% | 23.0% | 37.2% | 37.8% | 35.6% |
+| Spatial status | 86.1% | 70.1% | 79.8% | 85.3% | 82.8% |
+| Precise target retained | 71.7% | 60.7% | 73.6% | 71.2% | 68.8% |
+| Point within 100 | 38.3% | 9.2% | 34.0% | 33.6% | 27.8% |
+| Coarse direction exact | 34.4% | 11.3% | 31.2% | 33.0% | 25.8% |
+| No-target discipline | 96.3% | 93.3% | 92.4% | 95.6% | 95.5% |
+
+### Cleaned held-out deployment audit
+
+![Cleaned test comparison](./assets/cleaned_test_comparison.svg)
+
+After training, a stricter held-out audit removed pointing rows whose targets fall below the model-visible size threshold, then ran every release artifact on the complete revised validation and test splits. Strict-cleaned rows use only accepted evidence. Coverage-matched rows add evidence-preserving questions on the same held-out images to restore the earlier route and pointing action/status mix; they do not recreate the earlier object-class histogram.
+
+#### Final held-out test
+
+##### Strict-cleaned evidence
+
+| Metric | Merged BF16 - Strict cleaned test | CUDA bnb8 - Strict cleaned test | CUDA bnb4 - Strict cleaned test | MLX 8-bit - Strict cleaned test | MLX 4-bit - Strict cleaned test |
+|---|---:|---:|---:|---:|---:|
+| Valid JSON | 99.8% | 99.6% | 89.3% | 100.0% | 100.0% |
+| Schema valid | 99.8% | 99.6% | 89.2% | 100.0% | 62.8% |
+| Route accuracy | 99.8% | 99.5% | 89.1% | 100.0% | 63.3% |
+| Reference text F1 | 60.9% | 47.4% | 53.5% | 60.0% | 57.2% |
+| Spatial status | 88.0% | 76.7% | 72.0% | 86.1% | 83.8% |
+| Coordinate quality | 89.7% | 84.0% | 78.9% | 88.4% | 86.1% |
+| Precise target retained | 65.2% | 52.6% | 60.3% | 64.8% | 61.3% |
+| Point within 100 | 42.6% | 9.4% | 34.3% | 39.5% | 31.4% |
+| No-target discipline | 96.5% | 97.1% | 85.1% | 95.1% | 93.4% |
+
+##### Coverage-matched evidence
+
+| Metric | Merged BF16 - Coverage-matched test | CUDA bnb8 - Coverage-matched test | CUDA bnb4 - Coverage-matched test | MLX 8-bit - Coverage-matched test | MLX 4-bit - Coverage-matched test |
+|---|---:|---:|---:|---:|---:|
+| Valid JSON | 99.9% | 99.6% | 89.4% | 100.0% | 100.0% |
+| Schema valid | 99.9% | 99.5% | 89.3% | 100.0% | 50.2% |
+| Route accuracy | 99.9% | 99.5% | 89.1% | 100.0% | 50.5% |
+| Reference text F1 | 64.3% | 50.3% | 56.6% | 63.4% | 60.6% |
+| Spatial status | 84.2% | 64.1% | 70.8% | 82.5% | 79.0% |
+| Coordinate quality | 84.9% | 77.6% | 74.7% | 83.6% | 80.9% |
+| Precise target retained | 63.5% | 49.9% | 60.0% | 62.8% | 59.7% |
+| Point within 100 | 42.8% | 7.8% | 34.4% | 39.2% | 31.4% |
+| No-target discipline | 96.4% | 97.6% | 84.1% | 95.2% | 93.1% |
+
+#### Validation
+
+##### Strict-cleaned evidence
+
+| Metric | Merged BF16 - Strict cleaned validation | CUDA bnb8 - Strict cleaned validation | CUDA bnb4 - Strict cleaned validation | MLX 8-bit - Strict cleaned validation | MLX 4-bit - Strict cleaned validation |
+|---|---:|---:|---:|---:|---:|
+| Valid JSON | 99.9% | 99.6% | 96.4% | 100.0% | 100.0% |
+| Schema valid | 99.9% | 99.5% | 96.1% | 100.0% | 64.3% |
+| Route accuracy | 99.9% | 99.5% | 96.0% | 100.0% | 64.6% |
+| Reference text F1 | 59.9% | 48.1% | 56.8% | 59.0% | 56.7% |
+| Spatial status | 89.6% | 80.1% | 80.2% | 88.5% | 86.3% |
+| Coordinate quality | 89.8% | 84.9% | 85.5% | 88.9% | 87.4% |
+| Precise target retained | 65.8% | 55.7% | 67.8% | 65.8% | 62.9% |
+| Point within 100 | 39.3% | 9.6% | 34.6% | 35.0% | 28.6% |
+| No-target discipline | 96.7% | 97.2% | 91.8% | 96.1% | 95.2% |
+
+##### Coverage-matched evidence
+
+| Metric | Merged BF16 - Coverage-matched validation | CUDA bnb8 - Coverage-matched validation | CUDA bnb4 - Coverage-matched validation | MLX 8-bit - Coverage-matched validation | MLX 4-bit - Coverage-matched validation |
+|---|---:|---:|---:|---:|---:|
+| Valid JSON | 99.9% | 99.6% | 96.3% | 100.0% | 100.0% |
+| Schema valid | 99.9% | 99.5% | 96.0% | 100.0% | 53.0% |
+| Route accuracy | 99.9% | 99.4% | 95.9% | 100.0% | 53.2% |
+| Reference text F1 | 62.6% | 50.3% | 59.4% | 61.8% | 59.5% |
+| Spatial status | 84.5% | 66.7% | 76.6% | 83.5% | 80.7% |
+| Coordinate quality | 84.5% | 78.7% | 80.1% | 83.4% | 81.7% |
+| Precise target retained | 60.5% | 49.7% | 62.1% | 60.7% | 58.8% |
+| Point within 100 | 38.4% | 8.1% | 34.0% | 33.9% | 28.0% |
+| No-target discipline | 96.1% | 97.4% | 90.0% | 95.5% | 94.1% |
+
+Validation supports comparison and model selection; test is the final held-out report. These automated scores measure contract and reference agreement, not flight safety.
+
+### Experimental spoken-query input
+
+![Typed versus spoken input](./assets/spoken_input_comparison.svg)
+
+| Metric | Typed question | Spoken question |
+|---|---:|---:|
+| Valid JSON | 99.9% | 100.0% |
+| Schema valid | 99.9% | 98.1% |
+| Route accuracy | 99.9% | 97.6% |
+| Caption / answer F1 | 39.3% | 39.2% |
+| Spatial status | 71.3% | 65.3% |
+| Precise target retained | 50.5% | 39.3% |
+| Point within 100 | 36.3% | 31.9% |
+| Coarse direction exact | 34.7% | 34.3% |
+| No-target discipline | 88.2% | 88.2% |
+
+The image, ordinary-English wording, accepted JSON target, route, and case identity are matched; only the question modality changes. This checkpoint was trained on typed questions, so spoken input is an experimental zero-shot capability boundary rather than parity-tested speech support. A future generation will train directly on paired image-plus-audio questions.
+
+The independent benchmark rows use identical cases. Higher is better. Reference-overlap and coordinate scores are regression signals, not physical-world safety evidence.
+<!-- MIRIL_RELEASE_METRICS_END -->
+
+These automated metrics are regression signals, not proof of physical-world safety. Aerial reasoning and pointing still require human review and scenario-specific testing.
+
+## Technical Report
+
+The methods report explains how detector annotations became image-question-JSON supervision, why the four response envelopes exist, how observed model failures became preference data, and how the evidence gate prevents partial evaluations from becoming release claims:
+
+**[Miril-DroneVLM-2B-2: Compiling Detector Annotations into a Typed Aerial VLM](./technical_report.pdf)**
+
+The report treats data design as the central engineering contribution and clearly separates verified dataset properties from model-quality and operational-safety claims.
+
+## Citation
+
+```bibtex
+@techreport{sturges2026mirildrone2b2,
+  title       = {Miril-DroneVLM-2B-2: Compiling Detector Annotations into a Typed Aerial VLM},
+  author      = {Sturges, Stephan},
+  institution = {Miril.ai},
+  year        = {2026},
+  url         = {https://huggingface.co/MirilAI/Miril-DroneVLM-2B-2/blob/main/technical_report.pdf}
+}
+```
+
+GPT-5.5 and GPT-5.6 Sol contributed to the research, engineering, analysis, and editing. Stephan Sturges directed the work and accepts responsibility for the report and release.
+
+## Deployment Variants
+
+- [Miril-DroneVLM-2B-2-bnb8](https://huggingface.co/MirilAI/Miril-DroneVLM-2B-2-bnb8): 8-bit CUDA deployment.
+- [Miril-DroneVLM-2B-2-bnb4](https://huggingface.co/MirilAI/Miril-DroneVLM-2B-2-bnb4): 4-bit CUDA deployment.
+- [Miril-DroneVLM-2B-2-MLX-8bit](https://huggingface.co/MirilAI/Miril-DroneVLM-2B-2-MLX-8bit): 8-bit Apple Silicon deployment.
+- [Miril-DroneVLM-2B-2-MLX-4bit](https://huggingface.co/MirilAI/Miril-DroneVLM-2B-2-MLX-4bit): compact Apple Silicon deployment with a documented strict-schema routing regression.
+
+Variant cards report artifact size, recommended deployment memory, and
+complete same-case benchmark deltas.
+
+Treat those measured deltas as authoritative. The CUDA 8-bit and 4-bit
+packages use different bitsandbytes algorithms, so bit width alone does not
+rank their multimodal fidelity.
+
+The MLX 4-bit artifact remains available as a measured compact variant, not as an equivalent replacement for MLX 8-bit. On the complete balanced independent benchmark it preserves valid JSON at 100.0% after rounding but drops to 55.8% exact schema validity and 56.9% route accuracy, predominantly because pointing responses use `intent` where the contract requires `action`. The variant card contains the route-level evidence, operational warning, and the mixed 4/8-bit experiment planned for generation three.
+
+## WALDO Lineage
+
+Miril-DroneVLM-2B-2 is a separate VLM, but its aerial vocabulary grows from the [WALDO30 overhead detector](https://huggingface.co/StephanST/WALDO30). The two systems are complementary: WALDO supplies fast detection boxes; Miril-DroneVLM-2B-2 supplies language reasoning and typed responses.
+
+During dataset construction, WALDO annotations are treated as authoritative evidence for class presence, counts, and geometry after audit. A broad class name is not treated as proof of a narrower noun: an `UPole` box may contain a utility pole, antenna, dish, mast, or related fixture, and a `Boat` box may contain different kinds of watercraft. Narrow language is used only when reviewed image evidence supports it. This preserves the geometric value of the detector labels without asking the model to memorize invented detail.
+
+| WALDO class | What the class covers |
+| --- | --- |
+| `LightVehicle` | Cars, sedans, rickshaws, vans, and similar small personal road vehicles. Trucks and buses remain separate classes. |
+| `Person` | People and human figures visible from above. |
+| `Building` | Buildings and roofed structures, including sheds, shacks, pergolas, canopies, sunshades, and tents when supported by the image. |
+| `UPole` | Utility and telegraph poles, power-line fixtures, lamp posts, street fixtures, satellite dishes, antennas, and thin truss structures. |
+| `Boat` | Boats and watercraft such as ships, canoes, kayaks, and sailboats; surfboards may also occur in the source ontology. |
+| `Bike` | Bicycles, motorbikes, mopeds, scooters, and other two-wheeled vehicles. |
+| `Container` | Storage and shipping containers. |
+| `Truck` | Trucks, lorries, transport vehicles, semitrailers, articulated trucks, flatbeds, and larger industrial road vehicles. |
+| `Gastank` | Storage tanks, including liquid-storage and cylindrical tanks. |
+| `Digger` | Diggers, excavators, tractors when explicitly supported, and other construction vehicles or equipment. |
+| `Solarpanels` | Individual solar panels and panel arrays. |
+| `Bus` | Buses and bus-like passenger vehicles. |
+
+The VLM receives no hidden detector boxes or object-count metadata during inference. It sees only the image, fixed router system message, and user question.
+
+## Model Scope
+
+This checkpoint is derived from `google/gemma-4-E2B-it` and distributed as merged weights. Its aerial fine-tuning used drone-view images with typed English questions and the typed JSON contract above.
+
+The inherited Gemma architecture also accepts audio. We evaluate image-plus-spoken-question inference separately from matched typed questions and report that result as an experimental zero-shot capability, not as speech-training parity. Native temporal video understanding is not part of this release contract: the video demo samples individual frames and uses OpenCV between model calls.
+
+## Limits And Safety
+
+Miril-DroneVLM-2B-2 is a perception and reasoning model. It is not a flight controller, certified detect-and-avoid system, landing certifier, or standalone safety system.
+
+It can miss small people or objects, confuse similar classes, misread scale, return an incorrect answer, select an unsuitable region, or produce invalid JSON. Performance varies with altitude, compression, motion blur, weather, camera angle, and domain shift.
+
+For physical deployment:
+
+- validate the exact camera, altitude range, geography, and operating conditions,
+- independently check terrain, obstacles, people, vehicles, trees, wires, weather, and maps,
+- keep geofencing and vehicle constraints outside the model,
+- reject invalid JSON and non-target coordinates,
+- treat coarse directions as review cues only,
+- keep a human or separately validated planner in the decision loop,
+- log prompts, outputs, validation failures, and downstream actions.
+
+## Intended Use
+
+Good research and prototyping fits include:
+
+- searching and summarizing aerial imagery for human review,
+- inspection triage for infrastructure, agriculture, construction, and mapping,
+- first-response and disaster-recovery situational awareness,
+- delivery or landing-area pre-screening with explicit abstention,
+- object localization and tracker initialization under application control, and
+- typed-output experiments for physical AI.
+
+Do not use the model for direct unsupervised flight control, certified safety decisions, identifying people, or safety-critical operation without independent sensing and validation.
+
+## License
+
+Apache License 2.0. See [`LICENSE`](./LICENSE) and [`NOTICE`](./NOTICE).
+
+This model is derived from `google/gemma-4-E2B-it`. Review the upstream model documentation and applicable terms before deployment.

chat_template.jinja

@@ -0,0 +1,386 @@
+{#
+  Template: Google Gemma 4 Canonical Chat Template
+  Author: Google Gemma Engineering Team
+  Published: 2026-07-09
+  Context: Fixed tool-calling loops, turn closures, and thinking content-ordering.
+#}
+{%- macro format_parameters(properties, required, filter_keys=false) -%}
+    {%- set standard_keys = ['description', 'type', 'properties', 'required', 'nullable'] -%}
+    {%- set ns = namespace(found_first=false) -%}
+    {%- for key, value in properties | dictsort -%}
+        {%- set add_comma = false -%}
+        {%- if not filter_keys or key not in standard_keys -%}
+            {%- if ns.found_first %},{% endif -%}
+            {%- set ns.found_first = true -%}
+            {{ key }}:{
+            {%- if value['description'] -%}
+                description:<|"|>{{ value['description'] }}<|"|>
+                {%- set add_comma = true -%}
+            {%- endif -%}
+            {%- if value['type'] | upper == 'STRING' -%}
+                {%- if value['enum'] -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    enum:{{ format_argument(value['enum']) }}
+                {%- endif -%}
+            {%- elif value['type'] | upper == 'ARRAY' -%}
+                {%- if value['items'] is mapping and value['items'] -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    items:{
+                    {%- set ns_items = namespace(found_first=false) -%}
+                    {%- for item_key, item_value in value['items'] | dictsort -%}
+                        {%- if item_value is not none -%}
+                            {%- if ns_items.found_first %},{% endif -%}
+                            {%- set ns_items.found_first = true -%}
+                            {%- if item_key == 'properties' -%}
+                                properties:{
+                                {%- if item_value is mapping -%}
+                                    {{- format_parameters(item_value, value['items']['required'] | default([])) -}}
+                                {%- endif -%}
+                                }
+                            {%- elif item_key == 'required' -%}
+                                required:[
+                                {%- for req_item in item_value -%}
+                                    <|"|>{{- req_item -}}<|"|>
+                                    {%- if not loop.last %},{% endif -%}
+                                {%- endfor -%}
+                                ]
+                            {%- elif item_key == 'type' -%}
+                                {%- if item_value is string -%}
+                                    type:{{ format_argument(item_value | upper) }}
+                                {%- else -%}
+                                    type:{{ format_argument(item_value | map('upper') | list) }}
+                                {%- endif -%}
+                            {%- else -%}
+                                {{ item_key }}:{{ format_argument(item_value) }}
+                            {%- endif -%}
+                        {%- endif -%}
+                    {%- endfor -%}
+                    }
+                {%- endif -%}
+            {%- endif -%}
+            {%- if value['nullable'] %}
+                {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                nullable:true
+            {%- endif -%}
+            {%- if value['type'] | upper == 'OBJECT' -%}
+                {%- if value['properties'] is defined and value['properties'] is mapping -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    properties:{
+                    {{- format_parameters(value['properties'], value['required'] | default([])) -}}
+                    }
+                {%- elif value is mapping -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    properties:{
+                    {{- format_parameters(value, value['required'] | default([]), filter_keys=true) -}}
+                    }
+                {%- endif -%}
+                {%- if value['required'] -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    required:[
+                    {%- for item in value['required'] | default([]) -%}
+                        <|"|>{{- item -}}<|"|>
+                        {%- if not loop.last %},{% endif -%}
+                    {%- endfor -%}
+                    ]
+                {%- endif -%}
+            {%- endif -%}
+            {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+            type:<|"|>{{ value['type'] | upper }}<|"|>}
+        {%- endif -%}
+    {%- endfor -%}
+{%- endmacro -%}
+{%- macro format_function_declaration(tool_data) -%}
+    declaration:{{- tool_data['function']['name'] -}}{description:<|"|>{{- tool_data['function']['description'] -}}<|"|>
+    {%- set params = tool_data['function']['parameters'] -%}
+    {%- if params -%}
+        ,parameters:{
+        {%- if params['properties'] -%}
+            properties:{ {{- format_parameters(params['properties'], params['required']) -}} },
+        {%- endif -%}
+        {%- if params['required'] -%}
+            required:[
+            {%- for item in params['required'] -%}
+                <|"|>{{- item -}}<|"|>
+                {{- ',' if not loop.last -}}
+            {%- endfor -%}
+            ],
+        {%- endif -%}
+        {%- if params['type'] -%}
+            type:<|"|>{{- params['type'] | upper -}}<|"|>}
+        {%- endif -%}
+    {%- endif -%}
+    {%- if 'response' in tool_data['function'] -%}
+        {%- set response_declaration = tool_data['function']['response'] -%}
+        ,response:{
+        {%- if response_declaration['description'] -%}
+            description:<|"|>{{- response_declaration['description'] -}}<|"|>,
+        {%- endif -%}
+        {%- if response_declaration['type'] | upper == 'OBJECT' -%}
+            type:<|"|>{{- response_declaration['type'] | upper -}}<|"|>}
+        {%- endif -%}
+    {%- endif -%}
+    }
+{%- endmacro -%}
+{%- macro format_argument(argument, escape_keys=True) -%}
+    {%- if argument is none -%}
+        {{- 'null' -}}
+    {%- elif argument is string -%}
+        {{- '<|"|>' + argument + '<|"|>' -}}
+    {%- elif argument is boolean -%}
+        {{- 'true' if argument else 'false' -}}
+    {%- elif argument is mapping -%}
+        {{- '{' -}}
+        {%- set ns = namespace(found_first=false) -%}
+        {%- for key, value in argument | dictsort -%}
+            {%- if ns.found_first %},{% endif -%}
+            {%- set ns.found_first = true -%}
+            {%- if escape_keys -%}
+                {{- '<|"|>' + key + '<|"|>' -}}
+            {%- else -%}
+                {{- key -}}
+            {%- endif -%}
+            :{{- format_argument(value, escape_keys=escape_keys) -}}
+        {%- endfor -%}
+        {{- '}' -}}
+    {%- elif argument is sequence -%}
+        {{- '[' -}}
+        {%- for item in argument -%}
+            {{- format_argument(item, escape_keys=escape_keys) -}}
+            {%- if not loop.last %},{% endif -%}
+        {%- endfor -%}
+        {{- ']' -}}
+    {%- else -%}
+        {{- argument -}}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro strip_thinking(text) -%}
+    {%- set ns = namespace(result='') -%}
+    {%- for part in text.split('<channel|>') -%}
+        {%- if '<|channel>' in part -%}
+            {%- set ns.result = ns.result + part.split('<|channel>')[0] -%}
+        {%- else -%}
+            {%- set ns.result = ns.result + part -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- ns.result | trim -}}
+{%- endmacro -%}
+
+{%- macro format_tool_response_block(tool_name, response) -%}
+    {{- '<|tool_response>' -}}
+    {%- if response is mapping -%}
+        {{- 'response:' + tool_name + '{' -}}
+        {%- for key, value in response | dictsort -%}
+            {{- key -}}:{{- format_argument(value, escape_keys=False) -}}
+            {%- if not loop.last %},{% endif -%}
+        {%- endfor -%}
+        {{- '}' -}}
+    {%- else -%}
+        {{- 'response:' + tool_name + '{value:' + format_argument(response, escape_keys=False) + '}' -}}
+    {%- endif -%}
+    {{- '<tool_response|>' -}}
+{%- endmacro -%}
+
+{#- ===== SETUP ===== -#}
+{%- set ns = namespace(prev_message_type=None, prev_non_tool_role=None) -%}
+{%- set loop_messages = messages -%}
+{%- set enable_thinking = enable_thinking | default(false) -%}
+{%- set preserve_thinking = preserve_thinking | default(false) -%}
+{{- bos_token -}}
+{#- Handle System/Tool Definitions Block -#}
+{%- if enable_thinking or tools or (messages and messages[0]['role'] in ['system', 'developer']) -%}
+    {{- '<|turn>system\n' -}}
+    {#- Inject Thinking token at the very top of the FIRST system turn -#}
+    {%- if enable_thinking -%}
+        {{- '<|think|>\n' -}}
+        {%- set ns.prev_message_type = 'think' -%}
+    {%- endif -%}
+    {%- if messages and messages[0]['role'] in ['system', 'developer'] -%}
+        {%- if messages[0]['content'] is string -%}
+            {{- messages[0]['content'] | trim -}}
+        {%- elif messages[0]['content'] is sequence -%}
+            {%- for item in messages[0]['content'] -%}
+                {{- item['text'] | trim + ' '-}}
+            {%- endfor -%}
+        {%- endif -%}
+        {%- set loop_messages = messages[1:] -%}
+    {%- endif -%}
+    {%- if tools -%}
+        {%- for tool in tools %}
+            {{- '<|tool>' -}}
+            {{- format_function_declaration(tool) | trim -}}
+            {{- '<tool|>' -}}
+        {%- endfor %}
+        {%- set ns.prev_message_type = 'tool' -%}
+    {%- endif -%}
+    {{- '<turn|>\n' -}}
+{%- endif %}
+
+{#- Pre-scan: find last user message index for reasoning guard -#}
+{%- set ns_turn = namespace(last_user_idx=-1) -%}
+{%- for i in range(loop_messages | length) -%}
+    {%- if loop_messages[i]['role'] == 'user' -%}
+        {%- set ns_turn.last_user_idx = i -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- Loop through messages -#}
+{%- for message in loop_messages -%}
+    {%- if message['role'] != 'tool' -%}
+    {%- set ns.prev_message_type = None -%}
+    {%- set role = 'model' if message['role'] == 'assistant' else message['role'] -%}
+    {#- Detect continuation using tracked state — O(1) instead of O(n) backward scan -#}
+    {%- set continue_same_model_turn = (role == 'model' and ns.prev_non_tool_role == 'assistant') -%}
+    {%- if not continue_same_model_turn -%}
+        {{- '<|turn>' + role + '\n' }}
+    {%- endif -%}
+
+    {#- Render reasoning/reasoning_content as thinking channel -#}
+    {%- set thinking_text = message.get('reasoning') or message.get('reasoning_content') -%}
+    {%- set thinking_gate = (loop.index0 > ns_turn.last_user_idx) or (preserve_thinking and message.get('tool_calls')) -%}
+    {%- if thinking_text and thinking_gate -%}
+        {{- '<|channel>thought\n' + thinking_text + '\n<channel|>' -}}
+    {%- endif -%}
+
+            {%- if message.get('tool_calls') -%}
+                {%- for tool_call in message.get('tool_calls') -%}
+                    {%- set function = tool_call['function'] -%}
+                    {{- '<|tool_call>call:' + function['name'] + '{' -}}
+                    {%- if function['arguments'] is mapping -%}
+                        {%- set ns_args = namespace(found_first=false) -%}
+                        {%- for key, value in function['arguments'] | dictsort -%}
+                            {%- if ns_args.found_first %},{% endif -%}
+                            {%- set ns_args.found_first = true -%}
+                            {{- key -}}:{{- format_argument(value, escape_keys=False) -}}
+                        {%- endfor -%}
+                    {%- elif function['arguments'] is none -%}
+                    {%- else -%}
+                        {{- raise_exception(
+                            "chat_template: tool_calls[].function.arguments must be a "
+                            "JSON object (mapping), not a string. Deserialize arguments "
+                            "before passing to the template."
+                        ) -}}
+                    {%- endif -%}
+                    {{- '}<tool_call|>' -}}
+                {%- endfor -%}
+                {%- set ns.prev_message_type = 'tool_call' -%}
+            {%- endif -%}
+
+            {%- set ns_tr_out = namespace(flag=false) -%}
+            {%- if message.get('tool_responses') -%}
+                {#- Legacy: tool_responses embedded on the assistant message (Google/Gemma native) -#}
+                {%- for tool_response in message.get('tool_responses') -%}
+                    {{- format_tool_response_block(tool_response['name'] | default('unknown', true), tool_response['response']) -}}
+                    {%- set ns_tr_out.flag = true -%}
+                    {%- set ns.prev_message_type = 'tool_response' -%}
+                {%- endfor -%}
+            {%- elif message.get('tool_calls') -%}
+                {#- OpenAI Chat Completions: forward-scan consecutive role:tool messages -#}
+                {%- set ns_tool_scan = namespace(stopped=false) -%}
+                {%- for k in range(loop.index0 + 1, loop_messages | length) -%}
+                    {%- if ns_tool_scan.stopped -%}
+                    {%- elif loop_messages[k]['role'] != 'tool' -%}
+                        {%- set ns_tool_scan.stopped = true -%}
+                    {%- else -%}
+                        {%- set follow = loop_messages[k] -%}
+                        {#- Resolve tool_call_id to function name -#}
+                        {%- set ns_tname = namespace(name=follow.get('name') or 'unknown') -%}
+                        {%- for tc in message.get('tool_calls') -%}
+                            {%- if tc.get('id') == follow.get('tool_call_id') -%}
+                                {%- set ns_tname.name = tc['function']['name'] -%}
+                            {%- endif -%}
+                        {%- endfor -%}
+                        {#- Handle content as string or content-parts array -#}
+                        {%- set tool_body = follow.get('content') -%}
+                        {%- if tool_body is string -%}
+                            {{- format_tool_response_block(ns_tname.name, tool_body) -}}
+                        {%- elif tool_body is sequence and tool_body is not string -%}
+                            {%- set ns_txt = namespace(s='') -%}
+                            {%- for part in tool_body -%}
+                                {%- if part.get('type') == 'text' -%}
+                                    {%- set ns_txt.s = ns_txt.s + (part.get('text') | default('')) -%}
+                                {%- endif -%}
+                            {%- endfor -%}
+                            {{- format_tool_response_block(ns_tname.name, ns_txt.s) -}}
+                            {%- for part in tool_body -%}
+                                {%- if part.get('type') in ['image', 'image_url'] -%}
+                                    {{- '<|image|>' -}}
+                                {%- elif part.get('type') in ['audio', 'input_audio'] -%}
+                                    {{- '<|audio|>' -}}
+                                {%- elif part.get('type') == 'video' -%}
+                                    {{- '<|video|>' -}}
+                                {%- endif -%}
+                            {%- endfor -%}
+                        {%- else -%}
+                            {{- format_tool_response_block(ns_tname.name, tool_body) -}}
+                        {%- endif -%}
+                        {%- set ns_tr_out.flag = true -%}
+                        {%- set ns.prev_message_type = 'tool_response' -%}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- endif -%}
+
+            {%- set captured_content -%}
+            {%- if message.get('content') is string -%}
+                {%- if role == 'model' -%}
+                    {{- strip_thinking(message['content']) -}}
+                {%- else -%}
+                    {{- message['content'] | trim -}}
+                {%- endif -%}
+            {%- elif message.get('content') is sequence -%}
+                {%- for item in message['content'] -%}
+                    {%- if item.get('type') == 'text' -%}
+                        {%- if role == 'model' -%}
+                            {{- strip_thinking(item['text']) -}}
+                        {%- else -%}
+                            {{- item['text'] | trim -}}
+                        {%- endif -%}
+                    {%- elif item.get('type') in ['image', 'image_url'] -%}
+                        {{- '<|image|>' -}}
+                    {%- elif item.get('type') in ['audio', 'input_audio'] -%}
+                        {{- '<|audio|>' -}}
+                    {%- elif item.get('type') == 'video' -%}
+                        {{- '<|video|>' -}}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- endif -%}
+            {%- endset -%}
+
+            {{- captured_content -}}
+            {%- set has_content = captured_content | trim | length > 0 -%}
+
+        {#- Forward-scan: find next non-tool message role for continuation detection -#}
+        {%- set next_nt = namespace(role=None, found=false) -%}
+        {%- for j in range(loop.index0 + 1, loop_messages | length) -%}
+            {%- if not next_nt.found -%}
+                {%- if loop_messages[j]['role'] != 'tool' -%}
+                    {%- set next_nt.role = loop_messages[j]['role'] -%}
+                    {%- set next_nt.found = true -%}
+                {%- endif -%}
+            {%- endif -%}
+        {%- endfor -%}
+
+        {%- set continues_into_next = (
+            role == 'model'
+            and next_nt.role == 'assistant'
+            and (not message.get('tool_calls') or ns_tr_out.flag)
+        ) -%}
+
+        {%- if ns.prev_message_type == 'tool_call' and not ns_tr_out.flag -%}
+            {{- '<|tool_response>' -}}
+        {%- elif continues_into_next -%}
+        {%- elif not (ns_tr_out.flag and not has_content and not next_nt.found) -%}
+            {{- '<turn|>\n' -}}
+        {%- endif -%}
+
+    {#- Track previous non-tool role for next iteration (avoids O(n) backward scan) -#}
+    {%- set ns.prev_non_tool_role = message['role'] -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+    {%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%}
+        {{- '<|turn>model\n' -}}
+    {%- elif ns.prev_message_type == 'tool_response' and enable_thinking -%}
+        {{- '<|channel>thought\n' -}}
+    {%- endif -%}
+{%- endif -%}

config.json

@@ -0,0 +1,191 @@
+{
+  "architectures": [
+    "Gemma4ForConditionalGeneration"
+  ],
+  "audio_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "attention_chunk_size": 12,
+    "attention_context_left": 13,
+    "attention_context_right": 0,
+    "attention_invalid_logits_value": -1000000000.0,
+    "attention_logit_cap": 50.0,
+    "chunk_size_feed_forward": 0,
+    "conv_kernel_size": 5,
+    "dtype": "bfloat16",
+    "gradient_clipping": 10000000000.0,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "model_type": "gemma4_audio",
+    "num_attention_heads": 8,
+    "num_hidden_layers": 12,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_proj_dims": 1536,
+    "problem_type": null,
+    "residual_weight": 0.5,
+    "return_dict": true,
+    "rms_norm_eps": 1e-06,
+    "subsampling_conv_channels": [
+      128,
+      32
+    ],
+    "use_clipped_linears": true
+  },
+  "audio_token_id": 258881,
+  "boa_token_id": 256000,
+  "boi_token_id": 255999,
+  "dtype": "bfloat16",
+  "eoa_token_id": 258883,
+  "eoa_token_index": 258883,
+  "eoi_token_id": 258882,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "image_token_id": 258880,
+  "initializer_range": 0.02,
+  "model_type": "gemma4",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attention_k_eq_v": false,
+    "bos_token_id": 2,
+    "dtype": "bfloat16",
+    "enable_moe_block": false,
+    "eos_token_id": 1,
+    "expert_intermediate_size": null,
+    "final_logit_softcapping": 30.0,
+    "global_head_dim": 512,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 1536,
+    "hidden_size_per_layer_input": 256,
+    "initializer_range": 0.02,
+    "intermediate_size": 6144,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma4_text",
+    "moe_intermediate_size": null,
+    "num_attention_heads": 8,
+    "num_experts": null,
+    "num_global_key_value_heads": null,
+    "num_hidden_layers": 35,
+    "num_key_value_heads": 1,
+    "num_kv_shared_layers": 20,
+    "pad_token_id": 0,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "partial_rotary_factor": 0.25,
+        "rope_theta": 1000000.0,
+        "rope_type": "proportional"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 512,
+    "tie_word_embeddings": true,
+    "top_k_experts": null,
+    "use_bidirectional_attention": null,
+    "use_cache": true,
+    "use_double_wide_mlp": true,
+    "vocab_size": 262144,
+    "vocab_size_per_layer_input": 262144
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.1",
+  "video_token_id": 258884,
+  "vision_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "chunk_size_feed_forward": 0,
+    "default_output_length": 280,
+    "dtype": "bfloat16",
+    "global_head_dim": 64,
+    "head_dim": 64,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 768,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "max_position_embeddings": 131072,
+    "model_type": "gemma4_vision",
+    "num_attention_heads": 12,
+    "num_hidden_layers": 16,
+    "num_key_value_heads": 12,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "position_embedding_size": 10240,
+    "problem_type": null,
+    "return_dict": true,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "rope_theta": 100.0,
+      "rope_type": "default"
+    },
+    "standardize": false,
+    "use_clipped_linears": true
+  },
+  "vision_soft_tokens_per_image": 280
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 2,
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106,
+    50
+  ],
+  "pad_token_id": 0,
+  "temperature": 1.0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "5.12.1"
+}

image_overlay.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+"""Render Miril-DroneVLM-2B-2 responses beside one image."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from PIL import Image
+
+from inference import load_model
+from video_overlay import find_visual_payload, infer_responses, initial_center, render
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--image", required=True)
+    parser.add_argument("--output", required=True)
+    parser.add_argument("--prompt", action="append", required=True)
+    parser.add_argument("--model-id", default="MirilAI/Miril-DroneVLM-2B-2")
+    parser.add_argument("--processor-id")
+    parser.add_argument("--max-new-tokens", type=int, default=384)
+    parser.add_argument("--load-4bit", action="store_true")
+    return parser.parse_args()
+
+
+def main() -> int:
+    args = parse_args()
+    prompts = [prompt.strip() for prompt in args.prompt if prompt.strip()][:3]
+    if not prompts:
+        raise ValueError("at least one non-empty --prompt is required")
+    model, processor = load_model(
+        args.model_id,
+        args.processor_id or args.model_id,
+        args.load_4bit,
+    )
+    image = Image.open(args.image).convert("RGB")
+    responses = infer_responses(
+        model,
+        processor,
+        image,
+        prompts,
+        args.max_new_tokens,
+    )
+    precise, coarse = find_visual_payload(responses)
+    center = initial_center(precise, image.width, image.height)
+    rendered = render(
+        image,
+        responses,
+        tracked_center=center,
+        precise_payload=precise,
+        coarse_payload=coarse,
+        reveal_index=12,
+    )
+    output = Path(args.output)
+    output.parent.mkdir(parents=True, exist_ok=True)
+    rendered.save(output)
+    output.with_suffix(output.suffix + ".json").write_text(
+        json.dumps(
+            {
+                "model_id": args.model_id,
+                "image": args.image,
+                "output": str(output),
+                "responses": [
+                    {
+                        "prompt": response.prompt,
+                        "payload": response.payload,
+                        "validation_errors": response.errors,
+                    }
+                    for response in responses
+                ],
+            },
+            indent=2,
+            ensure_ascii=False,
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

inference.py

@@ -0,0 +1,273 @@
+#!/usr/bin/env python3
+"""Single-image inference with strict Miril-DroneVLM-2B-2 response validation."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+from PIL import Image
+
+from router_contract import (
+    TYPED_JSON_ROUTER_SYSTEM_PROMPT,
+    drawable_point,
+    typed_response_errors,
+)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--image", required=True)
+    query = parser.add_mutually_exclusive_group(required=True)
+    query.add_argument("--prompt")
+    query.add_argument("--audio", help="Experimental spoken-question audio file.")
+    parser.add_argument(
+        "--audio-transcript",
+        help="Optional audit/display transcript; it is not supplied to the model.",
+    )
+    parser.add_argument("--audio-sample-rate", type=int, default=16000)
+    parser.add_argument("--max-audio-seconds", type=float, default=30.0)
+    parser.add_argument("--model-id", default="MirilAI/Miril-DroneVLM-2B-2")
+    parser.add_argument("--processor-id")
+    parser.add_argument("--max-new-tokens", type=int, default=384)
+    parser.add_argument("--load-4bit", action="store_true")
+    parser.add_argument("--output-json")
+    return parser.parse_args()
+
+
+def load_audio_mono_float32(
+    path: str | Path,
+    *,
+    sample_rate: int = 16000,
+    max_seconds: float = 30.0,
+) -> Any:
+    import numpy as np
+
+    audio_path = Path(path).expanduser()
+    if not audio_path.is_file():
+        raise FileNotFoundError(f"audio file does not exist: {audio_path}")
+    if sample_rate <= 0 or max_seconds <= 0:
+        raise ValueError("audio sample rate and maximum duration must be positive")
+    process = subprocess.run(
+        [
+            "ffmpeg",
+            "-hide_banner",
+            "-loglevel",
+            "error",
+            "-i",
+            str(audio_path),
+            "-f",
+            "f32le",
+            "-acodec",
+            "pcm_f32le",
+            "-ac",
+            "1",
+            "-ar",
+            str(sample_rate),
+            "pipe:1",
+        ],
+        check=True,
+        stdout=subprocess.PIPE,
+    )
+    waveform = np.frombuffer(process.stdout, dtype="<f4").copy()
+    if waveform.size == 0:
+        raise ValueError(f"audio file decoded to zero samples: {audio_path}")
+    max_samples = int(round(sample_rate * max_seconds))
+    if waveform.size > max_samples:
+        raise ValueError(
+            f"audio exceeds --max-audio-seconds ({waveform.size / sample_rate:.2f}s)"
+        )
+    return waveform
+
+
+def parse_bare_json(text: str) -> tuple[dict[str, Any] | None, list[str]]:
+    stripped = text.strip()
+    try:
+        payload = json.loads(stripped)
+    except Exception as exc:
+        return None, [f"invalid bare JSON: {exc}"]
+    if not isinstance(payload, dict):
+        return None, ["response must be a JSON object"]
+    errors = typed_response_errors(payload)
+    return (payload if not errors else None), errors
+
+
+def dispatch_summary(payload: dict[str, Any]) -> dict[str, Any]:
+    response_type = payload["type"]
+    summary: dict[str, Any] = {
+        "type": response_type,
+        "dispatch_key": response_type,
+        "text": payload.get("caption")
+        if response_type != "answer"
+        else payload["answer"],
+        "drawable": False,
+        "trackable": False,
+    }
+    if response_type == "location":
+        summary["dispatch_key"] = f"location:{payload['intent']}"
+    elif response_type == "pointing":
+        summary["dispatch_key"] = f"pointing:{payload['action']}"
+
+    point = drawable_point(payload)
+    if point is not None:
+        x, y, visual_mode = point
+        summary.update(
+            {
+                "drawable": True,
+                "visual_mode": visual_mode,
+                "normalized_xy": [x, y],
+                "trackable": visual_mode == "precise_target",
+            }
+        )
+    return summary
+
+
+def load_model(model_id: str, processor_id: str, load_4bit: bool) -> tuple[Any, Any]:
+    import torch
+    import transformers
+    from transformers import AutoProcessor
+
+    processor = AutoProcessor.from_pretrained(processor_id)
+    model_class = None
+    for class_name in (
+        "AutoModelForMultimodalLM",
+        "AutoModelForImageTextToText",
+        "AutoModelForVision2Seq",
+    ):
+        model_class = getattr(transformers, class_name, None)
+        if model_class is not None:
+            break
+    if model_class is None:
+        raise RuntimeError(
+            "Installed transformers has no supported multimodal auto-model class"
+        )
+
+    model_kwargs: dict[str, Any] = {
+        "device_map": "auto",
+        "dtype": torch.bfloat16 if torch.cuda.is_available() else torch.float32,
+    }
+    if load_4bit:
+        from transformers import BitsAndBytesConfig
+
+        model_kwargs["quantization_config"] = BitsAndBytesConfig(
+            load_in_4bit=True,
+            bnb_4bit_quant_type="nf4",
+            bnb_4bit_compute_dtype=torch.bfloat16,
+        )
+    model = model_class.from_pretrained(model_id, **model_kwargs)
+    model.eval()
+    return model, processor
+
+
+def generate(
+    model: Any,
+    processor: Any,
+    image: Image.Image,
+    prompt: str | None,
+    *,
+    audio_waveform: Any | None = None,
+    max_new_tokens: int,
+) -> str:
+    import torch
+
+    if (prompt is None) == (audio_waveform is None):
+        raise ValueError("provide exactly one of prompt or audio_waveform")
+    user_content: list[dict[str, Any]] = [{"type": "image", "image": image}]
+    if audio_waveform is not None:
+        user_content.append({"type": "audio", "audio": audio_waveform})
+    else:
+        user_content.append({"type": "text", "text": prompt})
+    messages = [
+        {
+            "role": "system",
+            "content": [{"type": "text", "text": TYPED_JSON_ROUTER_SYSTEM_PROMPT}],
+        },
+        {
+            "role": "user",
+            "content": user_content,
+        },
+    ]
+    rendered = processor.apply_chat_template(
+        messages,
+        tokenize=False,
+        add_generation_prompt=True,
+    )
+    processor_kwargs: dict[str, Any] = {
+        "text": [rendered],
+        "images": [[image]],
+        "return_tensors": "pt",
+        "padding": True,
+    }
+    if audio_waveform is not None:
+        processor_kwargs["audio"] = [audio_waveform]
+    inputs = processor(**processor_kwargs)
+    device = next(model.parameters()).device
+    inputs = inputs.to(device)
+    with torch.no_grad():
+        output = model.generate(
+            **inputs,
+            max_new_tokens=max_new_tokens,
+            do_sample=False,
+            num_beams=1,
+        )
+    generated = output[:, inputs["input_ids"].shape[-1] :]
+    return processor.batch_decode(generated, skip_special_tokens=True)[0].strip()
+
+
+def main() -> int:
+    args = parse_args()
+    if args.audio_transcript and not args.audio:
+        raise ValueError("--audio-transcript requires --audio")
+    processor_id = args.processor_id or args.model_id
+    model, processor = load_model(args.model_id, processor_id, args.load_4bit)
+    image = Image.open(args.image).convert("RGB")
+    audio_waveform = (
+        load_audio_mono_float32(
+            args.audio,
+            sample_rate=args.audio_sample_rate,
+            max_seconds=args.max_audio_seconds,
+        )
+        if args.audio
+        else None
+    )
+    raw = generate(
+        model,
+        processor,
+        image,
+        args.prompt,
+        audio_waveform=audio_waveform,
+        max_new_tokens=args.max_new_tokens,
+    )
+    payload, errors = parse_bare_json(raw)
+    result = {
+        "ok": payload is not None,
+        "model_id": args.model_id,
+        "input_modality": "image_audio" if args.audio else "image_text",
+        "prompt": args.prompt,
+        "audio_path": str(Path(args.audio).expanduser()) if args.audio else None,
+        "audio_transcript_for_audit_only": args.audio_transcript,
+        "raw": raw,
+        "payload": payload,
+        "validation_errors": errors,
+        "dispatch": dispatch_summary(payload) if payload is not None else None,
+    }
+    rendered = json.dumps(result, indent=2, ensure_ascii=False)
+    print(rendered)
+    if args.output_json:
+        output_path = Path(args.output_json)
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        output_path.write_text(rendered + "\n", encoding="utf-8")
+    if errors:
+        print(
+            "Response rejected; do not dispatch or draw coordinates.", file=sys.stderr
+        )
+        return 2
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

mlx_inference.py

@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+"""Apple Silicon inference for Miril-DroneVLM-2B-2 MLX variants."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from inference import dispatch_summary, parse_bare_json
+from router_contract import TYPED_JSON_ROUTER_SYSTEM_PROMPT
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--image", required=True)
+    parser.add_argument("--prompt", required=True)
+    parser.add_argument("--model-id", default="MirilAI/Miril-DroneVLM-2B-2-MLX-4bit")
+    parser.add_argument("--max-tokens", type=int, default=384)
+    parser.add_argument("--output-json")
+    return parser.parse_args()
+
+
+def main() -> int:
+    from mlx_vlm import generate, load
+
+    args = parse_args()
+    model, processor = load(args.model_id, strict=True)
+    messages = [
+        {
+            "role": "system",
+            "content": [{"type": "text", "text": TYPED_JSON_ROUTER_SYSTEM_PROMPT}],
+        },
+        {
+            "role": "user",
+            "content": [
+                {"type": "image"},
+                {"type": "text", "text": args.prompt},
+            ],
+        },
+    ]
+    rendered_prompt = processor.apply_chat_template(
+        messages, add_generation_prompt=True
+    )
+    generated = generate(
+        model,
+        processor,
+        prompt=rendered_prompt,
+        image=args.image,
+        max_tokens=args.max_tokens,
+        temperature=0.0,
+        verbose=False,
+    )
+    raw = getattr(generated, "text", str(generated)).strip()
+    payload, errors = parse_bare_json(raw)
+    result = {
+        "ok": payload is not None,
+        "model_id": args.model_id,
+        "prompt": args.prompt,
+        "raw": raw,
+        "payload": payload,
+        "validation_errors": errors,
+        "dispatch": dispatch_summary(payload) if payload is not None else None,
+    }
+    rendered = json.dumps(result, indent=2, ensure_ascii=False)
+    print(rendered)
+    if args.output_json:
+        output_path = Path(args.output_json)
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        output_path.write_text(rendered + "\n", encoding="utf-8")
+    return 0 if not errors else 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03e6df4f0e3b2c20a0234ca1911f5095407f8040ebb6a76d04dfa34104b05430
+size 1422130776

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18d62274c80f49cc4ce4eee3234f641b97e80d75128af376173153c97ac12f37
+size 4987105346

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a76df230ed93d375a03bed2549eadb1ac498c478414e4f04486157482299800
+size 3799615716

processor_config.json

@@ -0,0 +1,75 @@
+{
+  "audio_ms_per_token": 40,
+  "audio_seq_length": 750,
+  "feature_extractor": {
+    "dither": 0.0,
+    "feature_extractor_type": "Gemma4AudioFeatureExtractor",
+    "feature_size": 128,
+    "fft_length": 512,
+    "fft_overdrive": false,
+    "frame_length": 320,
+    "hop_length": 160,
+    "input_scale_factor": 1.0,
+    "max_frequency": 8000.0,
+    "mel_floor": 0.001,
+    "min_frequency": 0.0,
+    "padding_side": "right",
+    "padding_value": 0.0,
+    "per_bin_mean": null,
+    "per_bin_stddev": null,
+    "preemphasis": 0.0,
+    "preemphasis_htk_flavor": true,
+    "return_attention_mask": true,
+    "sampling_rate": 16000
+  },
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": false,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_processor_type": "Gemma4ImageProcessor",
+    "image_seq_length": 280,
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 280,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098
+  },
+  "image_seq_length": 280,
+  "processor_class": "Gemma4Processor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 70,
+    "num_frames": 32,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "video_processor_type": "Gemma4VideoProcessor"
+  }
+}

requirements.txt

@@ -0,0 +1,9 @@
+accelerate
+bitsandbytes
+huggingface_hub
+numpy
+opencv-python-headless
+pillow
+torch
+torchvision
+transformers>=5.12.1

router_contract.py

@@ -0,0 +1,257 @@
+"""Immutable public inference contract for Miril-DroneVLM-2B-2."""
+
+from __future__ import annotations
+
+from typing import Any
+
+TYPED_RESPONSE_CONTRACT = "typed_json_router_v2"
+
+TYPED_JSON_ROUTER_SYSTEM_PROMPT = (
+    "You are a visual assistant for overhead and drone imagery. Inspect the image, understand the "
+    "user's ordinary English request, and return exactly one valid bare JSON object. Never use "
+    "Markdown or add text outside the JSON object. Choose the response type from the result the user "
+    "is asking for, not from one keyword in isolation. "
+    '(1) Use {"type":"caption","caption":string} when the user wants a general description, '
+    "summary, caption, or open-ended read of the whole image. "
+    '(2) Use {"type":"answer","answer":string} when the requested result is words rather than one '
+    "image point. This includes counts, presence, appearance, color, activity, scene classification, "
+    "hazards, suitability, explanations, and prose descriptions of where a group or feature is. A "
+    "question about whether a named area is safe is an answer; a request to choose an area is a "
+    "location. A question containing 'where' is still an answer when it asks for a relative or general "
+    "location in words. "
+    '(3) Use type "location" with intent "landing" or "delivery" only when the user asks you to '
+    "choose an area for landing a drone or placing a delivery. Landing or delivery intent takes "
+    "priority even if the request also says show, mark, point, or pick. "
+    '(4) Use type "pointing" only when the user asks for one concrete image point. The point may '
+    "identify one visible object or instance, or it may be an explicitly requested representative "
+    "point such as the center of one visible group. Set action to "
+    '"locate" for find/where/show-location requests, "point" for '
+    'point/mark/highlight/select/pin requests, or "track" for track/follow/watch/keep-on/lock-on '
+    "requests. Look-at and focus-on requests for one target use locate. The "
+    "model supplies an initial point for track requests; a downstream application performs temporal "
+    "tracking. Requests about where several objects are concentrated or how two things are positioned "
+    "use type answer unless the user explicitly asks for one representative center point. If a point, "
+    "locate, or track request names multiple or ambiguous instances without identifying one instance "
+    "or one representative point, preserve the requested pointing action but return status "
+    "ambiguous_target with null "
+    "coordinates rather than choosing one arbitrarily. "
+    "If one request mixes output types, use this precedence: choose location for landing or delivery "
+    "selection; otherwise choose pointing when one concrete point is explicitly requested; otherwise "
+    "choose answer for a factual or verbal result; otherwise choose caption for a whole-scene read. "
+    "If the user genuinely asks for both a landing selection and a delivery selection, use the "
+    "selection requested first. If one target is assigned more than one pointing action, use track "
+    "before point and point before locate. If pointing actions name different targets, use that same "
+    "action precedence but return ambiguous_target with null coordinates rather than choosing one "
+    "target arbitrarily. "
+    "Do not invent extra keys to combine unrelated tasks. "
+    "Every location or pointing object must contain exactly type, its intent or action, caption, "
+    "coordinate_system, point_2d, point_semantics, pointing_mode, status, x, and y. Coordinates use "
+    '"gemma_relative_0_1000_yx": every non-null coordinate is a number from 0 through 1000, '
+    "point_2d is [y,x], and x and y repeat those values. Use "
+    'pointing_mode "precise_point" only for an image-grounded point. Use point_semantics '
+    '"specific_point" for a concrete target and "representative_point" only when the point represents '
+    "a broader visible target. A location response alone may use "
+    '"coarse_grid_direction" with status "coarse_direction" for a broad directional cue; x and y '
+    "must each be exactly 250, 500, or 750. A coarse cue is never a landing point, delivery point, "
+    "object point, or tracking target. When no reliable target or "
+    'direction applies, use pointing_mode "none", point_semantics "not_applicable", and null '
+    "coordinate_system, point_2d, x, and y. For location, use target_found for a selected precise "
+    "point, no_safe_area when no usable area exists, coarse_direction only for the broad grid cue "
+    "described above, or unknown when the image is insufficient. For pointing, use target_found "
+    "for one grounded target, no_target when the requested object is absent, no_matching_target when "
+    "a requested relation has no match, ambiguous_target when one target cannot be selected, or "
+    "unknown when the image is insufficient. Do not omit required keys or add extra keys."
+)
+
+COORDINATE_SYSTEM = "gemma_relative_0_1000_yx"
+GRID_VALUES = {250.0, 500.0, 750.0}
+LOCATION_INTENTS = {"landing", "delivery"}
+POINTING_ACTIONS = {"locate", "point", "track"}
+LOCATION_STATUSES = {"target_found", "coarse_direction", "no_safe_area", "unknown"}
+POINTING_STATUSES = {
+    "target_found",
+    "no_target",
+    "no_matching_target",
+    "ambiguous_target",
+    "unknown",
+}
+
+
+def typed_response_errors(payload: Any) -> list[str]:
+    """Return contract violations; an empty list means the payload is dispatch-safe."""
+
+    if not isinstance(payload, dict):
+        return ["response must be a JSON object"]
+    response_type = payload.get("type")
+    if response_type == "caption":
+        return _text_response_errors(payload, text_key="caption")
+    if response_type == "answer":
+        return _text_response_errors(payload, text_key="answer")
+    if response_type == "location":
+        return _spatial_response_errors(
+            payload,
+            discriminator_key="intent",
+            discriminator_values=LOCATION_INTENTS,
+            statuses=LOCATION_STATUSES,
+            allow_coarse=True,
+        )
+    if response_type == "pointing":
+        return _spatial_response_errors(
+            payload,
+            discriminator_key="action",
+            discriminator_values=POINTING_ACTIONS,
+            statuses=POINTING_STATUSES,
+            allow_coarse=False,
+        )
+    return ["type must be caption, answer, location, or pointing"]
+
+
+def _text_response_errors(payload: dict[str, Any], *, text_key: str) -> list[str]:
+    required = {"type", text_key}
+    errors: list[str] = []
+    if set(payload) != required:
+        errors.append(f"expected exactly {sorted(required)}, got {sorted(payload)}")
+    value = payload.get(text_key)
+    if not isinstance(value, str) or not value.strip():
+        errors.append(f"{text_key} must be a non-empty string")
+    return errors
+
+
+def _spatial_response_errors(
+    payload: dict[str, Any],
+    *,
+    discriminator_key: str,
+    discriminator_values: set[str],
+    statuses: set[str],
+    allow_coarse: bool,
+) -> list[str]:
+    required = {
+        "type",
+        discriminator_key,
+        "caption",
+        "coordinate_system",
+        "point_2d",
+        "point_semantics",
+        "pointing_mode",
+        "status",
+        "x",
+        "y",
+    }
+    errors: list[str] = []
+    if set(payload) != required:
+        errors.append(f"expected exactly {sorted(required)}, got {sorted(payload)}")
+    if payload.get(discriminator_key) not in discriminator_values:
+        errors.append(
+            f"{discriminator_key} must be one of {sorted(discriminator_values)}"
+        )
+    if not isinstance(payload.get("caption"), str) or not payload["caption"].strip():
+        errors.append("caption must be a non-empty string")
+    if payload.get("status") not in statuses:
+        errors.append(f"status must be one of {sorted(statuses)}")
+
+    mode = payload.get("pointing_mode")
+    if mode == "precise_point":
+        if payload.get("status") != "target_found":
+            errors.append("precise_point requires status target_found")
+        if payload.get("point_semantics") not in {
+            "specific_point",
+            "representative_point",
+        }:
+            errors.append(
+                "precise_point requires specific_point or representative_point semantics"
+            )
+        errors.extend(_coordinate_errors(payload))
+    elif mode == "coarse_grid_direction":
+        if not allow_coarse:
+            errors.append("pointing responses cannot use coarse_grid_direction")
+        if payload.get("status") != "coarse_direction":
+            errors.append("coarse_grid_direction requires status coarse_direction")
+        if payload.get("point_semantics") != "representative_point":
+            errors.append(
+                "coarse_grid_direction requires representative_point semantics"
+            )
+        errors.extend(_coordinate_errors(payload))
+        if _complete_coordinates(payload) and (
+            float(payload["x"]) not in GRID_VALUES
+            or float(payload["y"]) not in GRID_VALUES
+        ):
+            errors.append("coarse coordinates must use the 250/500/750 grid")
+    elif mode == "none":
+        if payload.get("point_semantics") != "not_applicable":
+            errors.append("none requires not_applicable semantics")
+        if not _null_coordinates(payload):
+            errors.append("none requires null coordinate_system, point_2d, x, and y")
+        if payload.get("status") in {"target_found", "coarse_direction"}:
+            errors.append("none cannot use a target-bearing status")
+    else:
+        allowed = ["precise_point", "none"]
+        if allow_coarse:
+            allowed.insert(1, "coarse_grid_direction")
+        errors.append(f"pointing_mode must be one of {allowed}")
+    return errors
+
+
+def _coordinate_errors(payload: dict[str, Any]) -> list[str]:
+    errors: list[str] = []
+    if payload.get("coordinate_system") != COORDINATE_SYSTEM:
+        errors.append(f"coordinate_system must be {COORDINATE_SYSTEM}")
+    point = payload.get("point_2d")
+    x = payload.get("x")
+    y = payload.get("y")
+    if not _coordinate(x) or not _coordinate(y):
+        errors.append("x and y must be numbers from 0 through 1000")
+    if (
+        not isinstance(point, list)
+        or len(point) != 2
+        or not all(_coordinate(v) for v in point)
+    ):
+        errors.append("point_2d must be [y, x] with values from 0 through 1000")
+    elif (
+        _coordinate(x)
+        and _coordinate(y)
+        and (float(point[0]) != float(y) or float(point[1]) != float(x))
+    ):
+        errors.append("point_2d must exactly duplicate [y, x]")
+    return errors
+
+
+def _coordinate(value: Any) -> bool:
+    return (
+        isinstance(value, (int, float))
+        and not isinstance(value, bool)
+        and 0 <= value <= 1000
+    )
+
+
+def _complete_coordinates(payload: dict[str, Any]) -> bool:
+    point = payload.get("point_2d")
+    return (
+        payload.get("coordinate_system") == COORDINATE_SYSTEM
+        and _coordinate(payload.get("x"))
+        and _coordinate(payload.get("y"))
+        and isinstance(point, list)
+        and len(point) == 2
+        and all(_coordinate(value) for value in point)
+        and float(point[0]) == float(payload["y"])
+        and float(point[1]) == float(payload["x"])
+    )
+
+
+def _null_coordinates(payload: dict[str, Any]) -> bool:
+    return all(
+        payload.get(key) is None for key in ("coordinate_system", "point_2d", "x", "y")
+    )
+
+
+def drawable_point(payload: Any) -> tuple[float, float, str] | None:
+    """Return normalized x/y plus visual mode only for a safe drawable response."""
+
+    if typed_response_errors(payload):
+        return None
+    if payload.get("type") not in {"location", "pointing"}:
+        return None
+    if payload["pointing_mode"] == "precise_point":
+        return float(payload["x"]), float(payload["y"]), "precise_target"
+    if payload["pointing_mode"] == "coarse_grid_direction":
+        return float(payload["x"]), float(payload["y"]), "coarse_direction"
+    return None

shared_kv_export_report.json

@@ -0,0 +1,458 @@
+{
+  "applicable": true,
+  "artifact": "hf_merged",
+  "checks": [
+    {
+      "expected_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14,
+        15,
+        16,
+        17,
+        18,
+        19,
+        20,
+        21,
+        22,
+        23,
+        24,
+        25,
+        26,
+        27,
+        28,
+        29,
+        30,
+        31,
+        32,
+        33,
+        34
+      ],
+      "missing_layer_ids": [],
+      "name": "language.q_proj",
+      "observed_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14,
+        15,
+        16,
+        17,
+        18,
+        19,
+        20,
+        21,
+        22,
+        23,
+        24,
+        25,
+        26,
+        27,
+        28,
+        29,
+        30,
+        31,
+        32,
+        33,
+        34
+      ],
+      "ok": true,
+      "unexpected_layer_ids": []
+    },
+    {
+      "expected_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14,
+        15,
+        16,
+        17,
+        18,
+        19,
+        20,
+        21,
+        22,
+        23,
+        24,
+        25,
+        26,
+        27,
+        28,
+        29,
+        30,
+        31,
+        32,
+        33,
+        34
+      ],
+      "missing_layer_ids": [],
+      "name": "language.o_proj",
+      "observed_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14,
+        15,
+        16,
+        17,
+        18,
+        19,
+        20,
+        21,
+        22,
+        23,
+        24,
+        25,
+        26,
+        27,
+        28,
+        29,
+        30,
+        31,
+        32,
+        33,
+        34
+      ],
+      "ok": true,
+      "unexpected_layer_ids": []
+    },
+    {
+      "expected_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14,
+        15,
+        16,
+        17,
+        18,
+        19,
+        20,
+        21,
+        22,
+        23,
+        24,
+        25,
+        26,
+        27,
+        28,
+        29,
+        30,
+        31,
+        32,
+        33,
+        34
+      ],
+      "missing_layer_ids": [],
+      "name": "language.q_norm",
+      "observed_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14,
+        15,
+        16,
+        17,
+        18,
+        19,
+        20,
+        21,
+        22,
+        23,
+        24,
+        25,
+        26,
+        27,
+        28,
+        29,
+        30,
+        31,
+        32,
+        33,
+        34
+      ],
+      "ok": true,
+      "unexpected_layer_ids": []
+    },
+    {
+      "expected_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14
+      ],
+      "missing_layer_ids": [],
+      "name": "language.k_proj",
+      "observed_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14
+      ],
+      "ok": true,
+      "unexpected_layer_ids": []
+    },
+    {
+      "expected_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14
+      ],
+      "missing_layer_ids": [],
+      "name": "language.v_proj",
+      "observed_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14
+      ],
+      "ok": true,
+      "unexpected_layer_ids": []
+    },
+    {
+      "expected_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14
+      ],
+      "missing_layer_ids": [],
+      "name": "language.k_norm",
+      "observed_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14
+      ],
+      "ok": true,
+      "unexpected_layer_ids": []
+    },
+    {
+      "expected_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14,
+        15
+      ],
+      "missing_layer_ids": [],
+      "name": "vision.k_norm",
+      "observed_layer_ids": [
+        0,
+        1,
+        2,
+        3,
+        4,
+        5,
+        6,
+        7,
+        8,
+        9,
+        10,
+        11,
+        12,
+        13,
+        14,
+        15
+      ],
+      "ok": true,
+      "unexpected_layer_ids": []
+    }
+  ],
+  "errors": [],
+  "expected_language_k_norm_count": 15,
+  "expected_vision_k_norm_count": 16,
+  "explanation": "Gemma 4 serializes K/V projections and k_norm only for the leading non-shared language layers. Later language layers reuse shared K/V states and must not be treated as missing tensors.",
+  "kv_owner_layer_ids": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14
+  ],
+  "kv_shared_layer_ids": [
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31,
+    32,
+    33,
+    34
+  ],
+  "model_type": "gemma4",
+  "num_hidden_layers": 35,
+  "num_kv_shared_layers": 20,
+  "ok": true,
+  "schema_version": 1,
+  "serialized_tensor_count": 1951,
+  "serialized_weight_source": "model.safetensors.index.json"
+}

technical_report.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2314e40d060b4386ad1d396a2bf92e313167bbf1250cdd81bd829a2c8d620848
+size 1266065

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f
+size 32169626

tokenizer_config.json

@@ -0,0 +1,142 @@
+{
+  "audio_token": "<|audio|>",
+  "backend": "tokenizers",
+  "boa_token": "<|audio>",
+  "boi_token": "<|image>",
+  "bos_token": "<bos>",
+  "eoa_token": "<audio|>",
+  "eoc_token": "<channel|>",
+  "eoi_token": "<image|>",
+  "eos_token": "<eos>",
+  "eot_token": "<turn|>",
+  "escape_token": "<|\"|>",
+  "etc_token": "<tool_call|>",
+  "etd_token": "<tool|>",
+  "etr_token": "<tool_response|>",
+  "extra_special_tokens": [
+    "<|video|>"
+  ],
+  "image_token": "<|image|>",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "audio_token": "<|audio|>",
+    "boa_token": "<|audio>",
+    "boi_token": "<|image>",
+    "eoa_token": "<audio|>",
+    "eoc_token": "<channel|>",
+    "eoi_token": "<image|>",
+    "eot_token": "<turn|>",
+    "escape_token": "<|\"|>",
+    "etc_token": "<tool_call|>",
+    "etd_token": "<tool|>",
+    "etr_token": "<tool_response|>",
+    "image_token": "<|image|>",
+    "soc_token": "<|channel>",
+    "sot_token": "<|turn>",
+    "stc_token": "<|tool_call>",
+    "std_token": "<|tool>",
+    "str_token": "<|tool_response>",
+    "think_token": "<|think|>"
+  },
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "processor_class": "Gemma4Processor",
+  "response_schema": {
+    "properties": {
+      "content": {
+        "type": "string"
+      },
+      "role": {
+        "const": "assistant"
+      },
+      "thinking": {
+        "type": "string"
+      },
+      "tool_calls": {
+        "items": {
+          "properties": {
+            "function": {
+              "properties": {
+                "arguments": {
+                  "additionalProperties": {},
+                  "type": "object",
+                  "x-parser": "gemma4-tool-call"
+                },
+                "name": {
+                  "type": "string"
+                }
+              },
+              "type": "object",
+              "x-regex": "call\\:(?P<name>\\w+)(?P<arguments>\\{.*\\})"
+            },
+            "type": {
+              "const": "function"
+            }
+          },
+          "type": "object"
+        },
+        "type": "array",
+        "x-regex-iterator": "<\\|tool_call>(.*?)<tool_call\\|>"
+      }
+    },
+    "type": "object",
+    "x-regex": "(\\<\\|channel\\>thought\\n(?P<thinking>.*?)\\<channel\\|\\>)?(?P<tool_calls>\\<\\|tool_call\\>.*\\<tool_call\\|\\>)?(?P<content>(?:(?!\\<turn\\|\\>)(?!\\<\\|tool_response\\>).)+)?(?:\\<turn\\|\\>|\\<\\|tool_response\\>)?"
+  },
+  "response_template": {
+    "defaults": {
+      "role": "assistant"
+    },
+    "fields": {
+      "content": {
+        "close": [
+          "<turn|>",
+          "<|tool_response>",
+          "<eos>"
+        ],
+        "content": "text"
+      },
+      "thinking": {
+        "close": "<channel|>",
+        "content": "text",
+        "open": "<|channel>thought\n"
+      },
+      "tool_calls": {
+        "close": "<tool_call|>",
+        "content": "json",
+        "content_args": {
+          "string_delims": [
+            [
+              "<|\"|>",
+              "<|\"|>"
+            ]
+          ],
+          "unquoted_keys": true
+        },
+        "open_pattern": "<\\|tool_call>call:(?P<name>\\w+)",
+        "repeats": true,
+        "transform": {
+          "function": {
+            "arguments": "{content}",
+            "name": "{name}"
+          },
+          "type": "function"
+        }
+      }
+    },
+    "start_anchor": [
+      "<|turn>model\n",
+      "<tool_response|>"
+    ]
+  },
+  "soc_token": "<|channel>",
+  "sot_token": "<|turn>",
+  "stc_token": "<|tool_call>",
+  "std_token": "<|tool>",
+  "str_token": "<|tool_response>",
+  "think_token": "<|think|>",
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>"
+}

video_overlay.py

@@ -0,0 +1,529 @@
+#!/usr/bin/env python3
+"""Render Miril-DroneVLM-2B-2 JSON over a video with safe pointing and OpenCV tracking."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shutil
+import subprocess
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import cv2
+import numpy as np
+from PIL import Image, ImageDraw, ImageFont
+
+from inference import generate, load_model, parse_bare_json
+from router_contract import drawable_point
+
+
+GREEN = (42, 255, 128)
+WHITE = (232, 238, 235)
+PANEL = (7, 10, 11)
+
+
+@dataclass
+class Response:
+    prompt: str
+    label: str
+    text: str
+    payload: dict[str, Any] | None
+    errors: list[str]
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--input", required=True)
+    parser.add_argument("--output", required=True)
+    parser.add_argument("--prompt", action="append", required=True)
+    parser.add_argument("--model-id", default="MirilAI/Miril-DroneVLM-2B-2")
+    parser.add_argument("--processor-id")
+    parser.add_argument("--interval-seconds", type=float, default=5.0)
+    parser.add_argument(
+        "--crop", choices=("native", "center-square"), default="center-square"
+    )
+    parser.add_argument("--crop-size", type=int, default=1000)
+    parser.add_argument("--max-seconds", type=float, default=0.0)
+    parser.add_argument("--max-new-tokens", type=int, default=384)
+    parser.add_argument("--load-4bit", action="store_true")
+    return parser.parse_args()
+
+
+def process_frame(frame: np.ndarray, crop: str, crop_size: int) -> Image.Image:
+    image = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
+    if crop == "native":
+        return image
+    width, height = image.size
+    side = min(width, height)
+    left = (width - side) // 2
+    top = (height - side) // 2
+    image = image.crop((left, top, left + side, top + side))
+    return image.resize((crop_size, crop_size), Image.Resampling.LANCZOS)
+
+
+def response_label(payload: dict[str, Any] | None, index: int) -> str:
+    if not payload:
+        return f"Prompt {index}"
+    response_type = payload.get("type")
+    if response_type == "caption":
+        return "Scene"
+    if response_type == "answer":
+        return "Answer"
+    if response_type == "location":
+        return f"Location/{payload.get('intent', 'unknown')}"
+    if response_type == "pointing":
+        return f"Pointing/{payload.get('action', 'unknown')}"
+    return f"Prompt {index}"
+
+
+def response_text(raw: str, payload: dict[str, Any] | None, errors: list[str]) -> str:
+    if errors or payload is None:
+        return f"REJECTED: {'; '.join(errors) or 'invalid JSON'}"
+    response_type = payload["type"]
+    if response_type == "caption":
+        return payload["caption"]
+    if response_type == "answer":
+        return payload["answer"]
+    point = drawable_point(payload)
+    if point is None:
+        return f"{payload['status']} / no point: {payload['caption']}"
+    x, y, mode = point
+    mode_text = "precise target" if mode == "precise_target" else "broad direction only"
+    return f"{payload['status']} / {mode_text} / yx=[{y:.0f}, {x:.0f}]: {payload['caption']}"
+
+
+def infer_responses(
+    model: Any,
+    processor: Any,
+    image: Image.Image,
+    prompts: list[str],
+    max_new_tokens: int,
+) -> list[Response]:
+    responses: list[Response] = []
+    for index, prompt in enumerate(prompts, start=1):
+        raw = generate(
+            model,
+            processor,
+            image,
+            prompt,
+            max_new_tokens=max_new_tokens,
+        )
+        payload, errors = parse_bare_json(raw)
+        responses.append(
+            Response(
+                prompt=prompt,
+                label=response_label(payload, index),
+                text=response_text(raw, payload, errors),
+                payload=payload,
+                errors=errors,
+            )
+        )
+    return responses
+
+
+def find_visual_payload(
+    responses: list[Response],
+) -> tuple[dict[str, Any] | None, dict[str, Any] | None]:
+    precise = None
+    coarse = None
+    for response in responses:
+        point = drawable_point(response.payload)
+        if point is None:
+            continue
+        if point[2] == "precise_target" and precise is None:
+            precise = response.payload
+        elif point[2] == "coarse_direction" and coarse is None:
+            coarse = response.payload
+    return precise, coarse
+
+
+def font(size: int, bold: bool = False) -> ImageFont.ImageFont:
+    names = (
+        "/usr/share/fonts/truetype/dejavu/DejaVuSansMono-Bold.ttf"
+        if bold
+        else "/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf",
+        "/Library/Fonts/Arial Bold.ttf" if bold else "/Library/Fonts/Arial.ttf",
+    )
+    for name in names:
+        try:
+            return ImageFont.truetype(name, size)
+        except Exception:
+            pass
+    return ImageFont.load_default()
+
+
+def wrap(
+    draw: ImageDraw.ImageDraw, text: str, face: ImageFont.ImageFont, width: int
+) -> list[str]:
+    words = text.split()
+    if not words:
+        return [""]
+    lines: list[str] = []
+    current = words[0]
+    for word in words[1:]:
+        candidate = f"{current} {word}"
+        if draw.textbbox((0, 0), candidate, font=face)[2] <= width:
+            current = candidate
+        else:
+            lines.append(current)
+            current = word
+    lines.append(current)
+    return lines
+
+
+def visible_text(text: str, reveal_index: int) -> str:
+    if reveal_index >= 12:
+        return text
+    words = text.split()
+    keep = max(1, (len(words) * (reveal_index + 1) + 11) // 12)
+    return " ".join(words[:keep])
+
+
+def layout_lines(
+    draw: ImageDraw.ImageDraw,
+    responses: list[Response],
+    panel_width: int,
+    height: int,
+) -> tuple[ImageFont.ImageFont, ImageFont.ImageFont, int]:
+    available_width = panel_width - 36
+    for size in range(19, 8, -1):
+        regular = font(size)
+        bold = font(size, bold=True)
+        line_height = size + 6
+        line_count = 0
+        for response in responses:
+            line_count += len(
+                wrap(
+                    draw,
+                    f"{response.label} query: {response.prompt}",
+                    regular,
+                    available_width,
+                )
+            )
+            line_count += len(
+                wrap(
+                    draw, f"{response.label}: {response.text}", regular, available_width
+                )
+            )
+            line_count += 1
+        if line_count * line_height <= height - 70:
+            return regular, bold, line_height
+    return font(9), font(9, bold=True), 15
+
+
+def draw_marker(
+    draw: ImageDraw.ImageDraw,
+    payload: dict[str, Any],
+    width: int,
+    height: int,
+    *,
+    tracked_center: tuple[float, float] | None,
+) -> None:
+    point = drawable_point(payload)
+    if point is None:
+        return
+    x, y, mode = point
+    px = int(round(x / 1000.0 * width))
+    py = int(round(y / 1000.0 * height))
+    if mode == "precise_target" and tracked_center is not None:
+        px, py = map(lambda value: int(round(value)), tracked_center)
+    if mode == "coarse_direction":
+        radius = max(40, min(width, height) // 11)
+        draw.ellipse(
+            (px - radius, py - radius, px + radius, py + radius),
+            fill=(0, 230, 120, 34),
+            outline=(0, 230, 120, 225),
+            width=3,
+        )
+        return
+    half = 25
+    radius = 31
+    draw.ellipse(
+        (px - radius, py - radius, px + radius, py + radius),
+        fill=(0, 230, 120, 40),
+        outline=(0, 230, 120, 230),
+        width=2,
+    )
+    draw.rectangle(
+        (px - half, py - half, px + half, py + half), outline=(0, 230, 120), width=2
+    )
+    draw.line((px - 10, py, px + 10, py), fill=(0, 230, 120), width=2)
+    draw.line((px, py - 10, px, py + 10), fill=(0, 230, 120), width=2)
+
+
+def render(
+    image: Image.Image,
+    responses: list[Response],
+    *,
+    tracked_center: tuple[float, float] | None,
+    precise_payload: dict[str, Any] | None,
+    coarse_payload: dict[str, Any] | None,
+    reveal_index: int,
+) -> Image.Image:
+    width, height = image.size
+    panel_width = width // 2
+    canvas = Image.new("RGB", (width + panel_width, height), PANEL)
+    layer = Image.new("RGBA", image.size, (0, 0, 0, 0))
+    marker_draw = ImageDraw.Draw(layer)
+    if precise_payload is not None and tracked_center is not None:
+        draw_marker(
+            marker_draw, precise_payload, width, height, tracked_center=tracked_center
+        )
+    elif coarse_payload is not None and reveal_index < 12:
+        draw_marker(marker_draw, coarse_payload, width, height, tracked_center=None)
+    canvas.paste(
+        Image.alpha_composite(image.convert("RGBA"), layer).convert("RGB"), (0, 0)
+    )
+
+    draw = ImageDraw.Draw(canvas)
+    regular, bold, line_height = layout_lines(draw, responses, panel_width, height)
+    max_width = panel_width - 36
+    y = 18
+    overflow = False
+    for response in responses:
+        blocks = (
+            (f"{response.label} query:", response.prompt),
+            (f"{response.label}:", visible_text(response.text, reveal_index)),
+        )
+        for prefix, body in blocks:
+            first = True
+            for line in wrap(draw, f"{prefix} {body}", regular, max_width):
+                if y + line_height > height - 48:
+                    draw.text((width + 18, y), "...", fill=WHITE, font=regular)
+                    overflow = True
+                    break
+                if first and line.startswith(prefix):
+                    draw.text((width + 18, y), prefix, fill=GREEN, font=bold)
+                    prefix_width = draw.textbbox((0, 0), prefix, font=bold)[2]
+                    draw.text(
+                        (width + 22 + prefix_width, y),
+                        line[len(prefix) :].lstrip(),
+                        fill=WHITE,
+                        font=regular,
+                    )
+                else:
+                    draw.text((width + 18, y), line, fill=WHITE, font=regular)
+                first = False
+                y += line_height
+            if overflow:
+                break
+        if overflow:
+            break
+        y += line_height
+    brand = "Miril-DroneVLM-2B-2 / miril.ai"
+    small = font(max(10, min(15, height // 55)))
+    brand_width = draw.textbbox((0, 0), brand, font=small)[2]
+    draw.text(
+        (width + panel_width - brand_width - 18, height - 30),
+        brand,
+        fill=WHITE,
+        font=small,
+    )
+    return canvas
+
+
+def initial_center(
+    payload: dict[str, Any] | None, width: int, height: int
+) -> tuple[float, float] | None:
+    point = drawable_point(payload)
+    if point is None or point[2] != "precise_target":
+        return None
+    center = (point[0] / 1000.0 * width, point[1] / 1000.0 * height)
+    return center if inside_margin(center, width, height) else None
+
+
+def inside_margin(
+    center: tuple[float, float], width: int, height: int, margin: int = 50
+) -> bool:
+    return margin < center[0] < width - margin and margin < center[1] < height - margin
+
+
+def track_center(
+    previous: Image.Image,
+    current: Image.Image,
+    center: tuple[float, float],
+) -> tuple[float, float] | None:
+    width, height = current.size
+    if not inside_margin(center, width, height):
+        return None
+    half = 50
+    left = int(round(center[0])) - half
+    top = int(round(center[1])) - half
+    right = left + 100
+    bottom = top + 100
+    if left < 0 or top < 0 or right > width or bottom > height:
+        return None
+    points = np.asarray(
+        [
+            [x, y]
+            for y in np.linspace(top + 12, bottom - 12, 4)
+            for x in np.linspace(left + 12, right - 12, 4)
+        ],
+        dtype=np.float32,
+    ).reshape(-1, 1, 2)
+    previous_gray = cv2.cvtColor(np.asarray(previous), cv2.COLOR_RGB2GRAY)
+    current_gray = cv2.cvtColor(np.asarray(current), cv2.COLOR_RGB2GRAY)
+    moved, status, _ = cv2.calcOpticalFlowPyrLK(
+        previous_gray, current_gray, points, None
+    )
+    if moved is None or status is None:
+        return None
+    mask = status.reshape(-1) == 1
+    if int(mask.sum()) < 4:
+        return None
+    delta = np.median(moved[mask].reshape(-1, 2) - points[mask].reshape(-1, 2), axis=0)
+    updated = (float(center[0] + delta[0]), float(center[1] + delta[1]))
+    return updated if inside_margin(updated, width, height) else None
+
+
+def mux_h264(silent_path: Path, source_path: Path, output_path: Path) -> None:
+    ffmpeg = shutil.which("ffmpeg")
+    if ffmpeg is None:
+        shutil.move(silent_path, output_path)
+        return
+    subprocess.run(
+        [
+            ffmpeg,
+            "-y",
+            "-i",
+            str(silent_path),
+            "-i",
+            str(source_path),
+            "-map",
+            "0:v:0",
+            "-map",
+            "1:a?",
+            "-c:v",
+            "libx264",
+            "-crf",
+            "20",
+            "-preset",
+            "medium",
+            "-c:a",
+            "aac",
+            "-shortest",
+            str(output_path),
+        ],
+        check=True,
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+    )
+
+
+def main() -> int:
+    args = parse_args()
+    prompts = [prompt.strip() for prompt in args.prompt if prompt.strip()][:3]
+    if not prompts:
+        raise ValueError("at least one non-empty --prompt is required")
+    if args.interval_seconds < 1 or args.interval_seconds > 10:
+        raise ValueError("--interval-seconds must be from 1 through 10")
+    source_path = Path(args.input)
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    model, processor = load_model(
+        args.model_id,
+        args.processor_id or args.model_id,
+        args.load_4bit,
+    )
+
+    capture = cv2.VideoCapture(str(source_path))
+    if not capture.isOpened():
+        raise ValueError(f"could not open {source_path}")
+    fps = float(capture.get(cv2.CAP_PROP_FPS) or 24.0)
+    max_frames = int(args.max_seconds * fps) if args.max_seconds > 0 else None
+    interval_frames = max(1, int(round(args.interval_seconds * fps)))
+    temporary_dir = Path(tempfile.mkdtemp(prefix="miril_drone_overlay_"))
+    silent_path = temporary_dir / "silent.mp4"
+    writer: Any = None
+    previous: Image.Image | None = None
+    responses: list[Response] = []
+    precise_payload = None
+    coarse_payload = None
+    center = None
+    segment_index = 0
+    frame_index = 0
+    sidecar_samples: list[dict[str, Any]] = []
+
+    while max_frames is None or frame_index < max_frames:
+        ok, frame = capture.read()
+        if not ok:
+            break
+        image = process_frame(frame, args.crop, args.crop_size)
+        if frame_index % interval_frames == 0 or not responses:
+            responses = infer_responses(
+                model,
+                processor,
+                image,
+                prompts,
+                args.max_new_tokens,
+            )
+            precise_payload, coarse_payload = find_visual_payload(responses)
+            center = initial_center(precise_payload, image.width, image.height)
+            segment_index = 0
+            sidecar_samples.append(
+                {
+                    "timestamp_s": round(frame_index / fps, 3),
+                    "responses": [
+                        {
+                            "prompt": response.prompt,
+                            "payload": response.payload,
+                            "validation_errors": response.errors,
+                        }
+                        for response in responses
+                    ],
+                }
+            )
+        elif center is not None and previous is not None:
+            center = track_center(previous, image, center)
+
+        rendered = render(
+            image,
+            responses,
+            tracked_center=center,
+            precise_payload=precise_payload,
+            coarse_payload=coarse_payload,
+            reveal_index=segment_index,
+        )
+        if writer is None:
+            writer = cv2.VideoWriter(
+                str(silent_path),
+                cv2.VideoWriter_fourcc(*"mp4v"),
+                fps,
+                rendered.size,
+            )
+        writer.write(cv2.cvtColor(np.asarray(rendered), cv2.COLOR_RGB2BGR))
+        previous = image
+        segment_index += 1
+        frame_index += 1
+
+    capture.release()
+    if writer is not None:
+        writer.release()
+    if frame_index == 0:
+        raise ValueError("no frames were rendered")
+    mux_h264(silent_path, source_path, output_path)
+    sidecar = {
+        "model_id": args.model_id,
+        "input": str(source_path),
+        "output": str(output_path),
+        "prompts": prompts,
+        "interval_seconds": args.interval_seconds,
+        "source_fps": fps,
+        "rendered_frames": frame_index,
+        "tracking_box_px": 100,
+        "edge_hide_margin_px": 50,
+        "samples": sidecar_samples,
+    }
+    output_path.with_suffix(output_path.suffix + ".json").write_text(
+        json.dumps(sidecar, indent=2, ensure_ascii=False) + "\n",
+        encoding="utf-8",
+    )
+    shutil.rmtree(temporary_dir, ignore_errors=True)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())