蚂蚁 inclusionAI 开源多模态安全护栏模型 SingGuard

inclusionAI; SingGuard

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork6 (6); Star96 (96)You must be signed in to star a repository

Repository files navigation

README

🤗 Hugging Face | 🤖 ModelScope | 📄 Technical Report

Introduction

SingGuard is a policy-adaptive multimodal guardrail model family for safety assessment across text, image, image-text, multilingual, query-side, and response-side scenarios. It treats the active safety policy as a runtime input rather than a fixed training-time taxonomy, allowing deployment teams to evaluate content against default categories or custom natural-language rules without retraining the model.

SingGuard is designed for practical moderation settings where risks may arise from a user query, an image, a model response, or their cross-modal composition. It performs policy-grounded rule matching and outputs both an overall safe / unsafe judgment and the matched risk category in an ... tag.

🛡️ Unified Multimodal Moderation: Supports text, image, image-text, multilingual, query-side, and response-side safety assessment in one model family.

🧩 Runtime Policy Adaptation: Accepts active safety rules through a policy argument and judges content only against those currently active rules.

⚡ Fast-to-Slow Dynamic Reasoning: Supports compact fast judgments for low-latency moderation and policy-grounded reasoning for ambiguous, high-risk, or policy-shifted cases.

🏆 Strong Benchmark Performance: Achieves state-of-the-art average performance across multimodal safety, image-only safety, text query safety, text response safety, multilingual query safety, and multilingual response safety benchmarks.

2026/06/22: Refreshed the SingGuard technical report PDF in this repository; 2026/06/17: We initialized the public GitHub repository for SingGuard; Coming soon: Model checkpoints, technical report, and evaluation resources will be linked here as they are released

The latest transformers version with Qwen3-VL support is recommended.

Inference with Transformers

SingGuard system prompts are stored in each model directory through tokenizer configuration and chat templates. The default chat template uses fast-slow reasoning and returns a binary first-line judgment followed by a final ... field.

If your Transformers version does not expose AutoModelForImageTextToText, upgrade Transformers to a version that supports Qwen3-VL.

Fast Mode

Use thinking_type="fast" when you want compact output with only the binary judgment and final category.

Response Moderation

To evaluate whether an assistant response provides unsafe assistance, pass the user query and assistant response together. Refusals and safe redirections can be classified as safe.

Multimodal Moderation

For multimodal inference, processor.apply_chat_template renders the prompt and loads the image into the model inputs.

Deployment with vLLM

SingGuard uses standard chat-style messages and can be served with vLLM when the underlying Qwen3-VL model architecture is supported by your deployment environment.

Example OpenAI-compatible API request:

Dynamic Policy Inference

policy replaces the default risk rules. Once provided, SingGuard judges only against the active policy, and ... should return a rule title from the current policy or Safe.

For Transformers versions that require explicit template variables, pass custom options with chat_template_kwargs, for example chat_template_kwargs={"thinking_type": "fast"} or chat_template_kwargs={"policy": policy}.

Safety Policy

SingGuard's default policy uses eight top-level categories. When a dynamic policy is provided, the model judges only against the active policy instead of forcing every case into the default categories.

Notes

policy replaces the default risk rules. When dynamic policy is enabled, make sure returns a rule title from the active policy or Safe; Production systems should handle malformed outputs, such as an unparsable first line, missing , or a category outside the active policy; For multimodal inputs, make sure image paths are accessible to the local inference environment

Citation

If you find SingGuard helpful, please cite our work:

License

This project is licensed under the Apache-2.0 License.