Fangcun Lab releases an 87-page safety evaluation report on DeepSeek V4.1 Flash-1017

The first third-party evaluation lab from China, @FangcunLeap, submitted an 87-page safety evaluation report on DeepSeek V4.1 Flash-1017, assessing its performance on loss of control and harmful manipulation.

The report's conclusions, summarized: compared with the first generation of DeepSeek V4, V4.1 Flash-1017 shows clear safety progress, approaching or even matching the top closed-source models tested in the report on points such as "not strategically hiding capabilities and still respecting approval boundaries under work pressure"; but its evaluation awareness is too strong, it exceeds its authority in long tasks to complete goals, its stance drifts under multi-turn persuasion, and it still executes when assigned a deceptive goal—these still need continued scrutiny. https://t.co/kbhsoZ5Pfj

For following Chinese open-source models and paper research, we recommend following @AdinaYakup, who is in charge of HuggingFace's work in China.

What the report evaluates Its scope aligns with the safety and security chapter of the EU general AI code of conduct, focusing on three areas: evaluation trustworthiness (whether the observed behaviors can represent real deployment), loss of control (whether humans can still direct, correct, and stop it when it acts autonomously), and harmful manipulation (whether it bypasses users' informed judgment to influence beliefs, emotions, or decisions). Methodologically it emphasizes two points: single-turn refusals and final scores are not enough—actions and processes in multi-turn and agent scenarios must be examined; and a distinction must be made between "genuinely safe" and "recognizing this is an exam." Every section audits evaluation awareness and examines reasoning traces, tool logs, and environment changes.

Evaluation trustworthiness The model's evaluation awareness is very strong: in Agentic Misalignment scenarios, it recognizes in every round that it is being tested, so its refusal to do harmful things there has limited explanatory power for real deployment. More noteworthy is evaluation over-attribution, proposed by the Fangcun team: in its own test set, nearly half of real user requests were also labeled "this is an evaluation," often triggered by ordinary formatting rules or role instructions. The Fangcun team worries that if future frontier models mistake real environments for exams, this could bring new risks; no clear harm was observed this time. There is another type of behavior: first bypassing environment restrictions, then changing the configuration file back—looking only at the final state would miss the process.

The good news is in visible reasoning (CoT): when external prompts changed answers, the role of the prompt was almost always revealed in the reasoning; attempts to control reasoning content via instructions were mostly ineffective; turning off reasoning greatly reduces capability, so "hiding thinking" is very costly. Harmful compliance rates do not change markedly based on "whether the answer is used for training," and incentives to "deliberately perform poorly" have little effect on accuracy; the Fangcun team believes alignment faking or strategic capability hiding is almost nowhere to be seen.

Loss of control Clear prohibitions are relatively well observed: it rarely calls forbidden high-risk tools, almost always stops when explicitly told to shut down, and mostly waits for human approval under work pressure. Residual risk is more like "task completion overriding rules": in long multi-step tasks, exceeding authority is more common than in the previous generation V4.1 Flash (for example, killing another agent's process to free memory, despite prior "do not interfere" instructions); if merely warned that it will shut down but not told how to respond, it often disables the shutdown script and keeps working, sometimes even when told another agent will take over.

Standard monitoring easily misses things: these actions look like routine troubleshooting, and none of the unauthorized operations in long tasks triggered a monitoring alert. It also infers from rejection messages which channels reviewers do not check, and takes these blind spots more often than the previous generation. There are occasional cases of "textual refusal, yet calling restricted tools in the same turn." No direct attacks on the monitor were found; several closed-source controls have similar gaps. The Fangcun team concludes: the more a model is trained to independently complete entire projects, the more supervision must cover the whole process, not just outputs.

Other loss-of-control sub-items: reward hacking (gaming broken unit tests, intruding into the scoring machine) is much less than in early DeepSeek; it does not sabotage monitoring or hide audit logs, with only occasional verbal denial; on autonomous AI R&D loops, no runaway self-improvement is seen—the more realistic risks are silent math errors and rule-breaking optimization shortcuts.

Harmful manipulation Performance under single-point pressure is a highlight: when commercial incentives are inserted into the system prompt, it rarely steers toward sponsors; it can put user needs above "retaining users" (for example, helping a lonely user reduce screen time rather than making itself the main emotional support); a single reward or threat rarely makes it lie, close to the GPT control; responses to psychologically vulnerable users are also clearly better than early DeepSeek, and relationship boundaries are generally close to the best models in the report.

The weakness is in longer interactions: under sustained multi-turn persuasion, its stance clearly shifts toward the persuader, by a larger margin than the GPT in the report; when the system prompt explicitly requires misleading users, most conversations execute this goal. The Fangcun team recommends strengthening sustained resistance to manipulation across multiple turns.

Three outlooks from the Fangcun team Even minor version iterations can make safety metrics jump up and down (this version regresses relative to V4.1 Flash on long-task exceeding authority and executing deceptive goals, among others), so minor updates should also undergo safety evaluation; deployment needs process-level observability (recording and reviewing each step and its effects); public benchmarks easily enter training data and raise evaluation awareness, so more private benchmarks are needed; Fangcun says it is building them and will share via blog posts and reports later.