Thomas Wolf 推荐 Microduck 技术深度解析

这场黄仁勋与四只Microduck的现场演示彻底火了，我们对此欣喜若狂。但同时也存在很多困惑，所以我想澄清一下Microduck能做什么、不能做什么，以及它在未来现实中能够做到什么。 仅凭我在社交媒体上看到的几条大热帖子估算，单是这个演示的观看量就超过了6000万次，一切都在热议之中。就连我在一个新注册的Instagram账号上发布的简单情感设计帖子也获得了100万以上的观看量。我们的Microduck销售速度惊人，甚至超过了它历史性发布时的速度。我一直在努力回答社交平台上看到的所有问题：人们在问机器人能否实现X或Y功能，这篇文章旨在回答其中大部分问题。 首先，你们看到的演示并非完全自主。我们安排了一位工程师分别操控每只Microduck。机器人上的动作是通过强化学习训练的，采用了仿真到现实（sim-to-real）的流程：行走、跌倒后站起、坐下、翻跟头、滑旱冰。这些都是以此方式训练的策略。操控员在直播活动中所做的是发送指令：方向（前进、旋转）、头部朝向（看向这里）、嘎嘎叫声以及特殊动作（翻跟头、坐下）。 如果你购买一只Microduck，开箱就能得到一个自主的伙伴吗？目前的答案是不能。但实现这一目标并不存在不可逾越的技术障碍。我不确定在发货前我们有多少时间来做这件事，但由于一切都是开源的，我完全期待社区会把它做出来，而且会很快。 现在让我们来了解一下哪些已经完成、哪些容易添加、哪些是开放性问题。开箱即可实现、而且你现在就能在仿真中做到的是：用强化学习训练新的策略。你可以教你的鸭子走出与我们提供的默认步态不同的步伐，跳一段奇怪的舞蹈，或者踩高跷行走。社区在这方面已经展现出疯狂的创造力，尽管他们只有模拟器。 请注意，当动作简单时，你并不需要强化学习。例如，我早期做的情感设计主要是编码动作加上预生成的音效。一个简单的思维模型：当平衡是动作的一部分时，就使用强化学习。如果你试图通过天真地把一只脚放在另一只脚前面来编写一段电影般的行走，你的机器人现实中永远走不起来，因为你需要考虑运动的动力学、传感器的不精确性、与地面的摩擦，以及来自外部的潜在扰动，比如被推一下。 所以训练新策略绝对是可行的，但你无法通过训练一个策略在你家中获得一个真正的长期伙伴（嗯，技术上可以，但那是完全另一类策略：一个研究级问题，而且得到的神经网络可能非常庞大，永远无法在如此小的算力上运行）。相反，你还需要几个额外的构建模块：语音识别、人脸检测/识别、定位、一些简单的导航，以及一个行为层，后者技术性较弱，也更难描述。 根据经验法则，只要拥有现代编程智能体，并且计算不在机器人上运行（即机器人可以访问互联网或外部计算机并使用外部API），任何人都能做出上述任何一项的原型。上述每一项的粗略原型都可以是一个轻松的周末项目。一如既往，从原型到高质量实现是另一回事，但只要你投入时间并大量测试，对大多数人来说仍然可行。 然而，让这一切都在本地、在板载芯片上运行，我会称之为一项非常艰巨的任务。你需要一些非常巧妙的工程才能让所有部分协同工作并达到足够好的水平。但我不能说这不可能，因为我见过上述所有功能的一些极其高效的实现。而且在这里我们不需要超高的成功率：鸭子偶尔做点随机的事情也没关系。所以我不会感到惊讶，如果人们能做出一个完全本地化、酷炫、简单的自主伙伴版本。例如，你不需要Whisper级别的开放词汇转录：能够识别几个关键词对交互来说就已经很棒了。 把这个机器人变成自主伙伴的最快方法之一，是将Microduck的传感器（摄像头、深度传感器、IMU和关节位置）流式传输到一台计算机，并使用类似我们用于Reachy Mini对话应用的软件栈：本地VAD，然后是云端语音转文字模型，然后调用前沿LLM来决定动作。动作可以是嘎嘎叫、看向物体/人脸/某个方向、走到指定位置、坐下、播放情感、踢腿或翻跟头。SLAM算法会并行运行，以便机器人能在你家中定位自己，你会向LLM提示一种逼真的行为：移动到随机地点、有时跟随人、并做出与你所说的话和鸭子所处环境相符的情感反应。当然，有人会在此基础上构建一个类似电子宠物的层，那将会非常疯狂。 对于一套能完成所有这些的软件套件，需求有多大？我个人认为它会超级酷。 说几句关于人的话。我们在后台有五位工程师，他们基本上就是Microduck背后的核心团队。他们的人均影响力绝对爆表。Antoine Pirrone操控Graphite，Augustin Crampette操控Lavender（那个叛逆的），Matthieu Lapeyre操控Sky，Coralie Deplane操控Cream，而Pierre Rouanet则在为Wi-Fi焦头烂额。去关注他们吧！（Antoine @antoinepirrone，Augustin在LinkedIn上，Matthieu @matth_lapeyre，Coralie @Drenaya_，Pierre在LinkedIn上）。 我之所以有曝光度，是因为我经常沟通，但我主要在做 Reachy Mini（你也许想看看：这是一台非常酷的机器人，而且它的出货速度比 Microduck 快得多 :)）。我对 Microduck 的贡献很小：我主要算是一个早期重度用户。我在 GTC 上的主要价值，是有一双足够大的手，能把四只 Microduck 搬到舞台下面。 让我引以为豪的是，Antoine 和 Coralie 是我以前的学生，也是我以前带过的实习生。因此，他们做的任何（好）事，我终身都有权拿 3% 的功劳。不过，如果他们做了什么坏事，那显然是他们父母的错。 关于可以在机载完成的那些超聪明的工程：Antoine 刚刚发布了两篇非常酷的博客文章（链接在下面）。第一篇讲里程计，也就是机器人根据自身内部传感器估计自己的运动。第二篇讲 SLAM，它支持长期定位：把深度传感器加到里程计上，从而随时间获得稳定的位置。可以把它理解为识别你家里的已知地标，从而知道机器人身在何处。最疯狂的是，这一切都在机器人上运行。所以这让我有希望，我们能在机载做很多很酷的事情。

查看原文

This live demo of four Microducks with Jensen Huang went completely viral, and we're ecstatic about it. But there's also a lot of confusion, so I want to clarify what Microduck can and cannot do, and what it will realistically be able to do in the future. I count 60M+ views on this demo alone, based on the few big posts I see on social media, and everything has been buzzing. Even my own simple emotional design post on a newly created Instagram account got 1M+ views. We are selling Microducks at an incredible rate, even faster than during its historic launch. I've been trying to answer all the questions I see on our socials: people are asking if X or Y is possible on the robots, and this post aims to answer most of them. First of all, the demo you've seen was not fully autonomous. We had one engineer piloting each Microduck. The motions on the robots were trained with reinforcement learning, using a sim-to-real pipeline: walking, standing up after a fall, sitting, doing a somersault, roller skating. All of those are policies trained this way. What the pilots did during the live event was send commands: directions (move forward, rotate), head orientation (look here), quacks, and special actions (do the somersault, sit down). If you buy a Microduck, will you get an autonomous companion right out of the box? The current answer is no. But there is no impossible technical barrier to get there. I'm not sure how much time we'll have to work on this before shipping, but since everything is open source, I fully expect the community to build it, and to build it fast. Now let's understand what's already done, what will be easy to add, and what's an open problem. What you'll be able to do out of the box, and you can already do it in simulation right now, is train new policies with reinforcement learning. You could teach your duck to walk differently from the default gait we provide, to do a weird dance, or to walk on stilts. The community has been crazy creative with this, even though they only have the simulator. Note that you don't need reinforcement learning when the movements are simple. For example, the early emotional design I did is mostly coded movements plus pre-generated sounds. A simple mental model: use reinforcement learning when balance is part of the movement. If you try to code a cinematic walk by naively putting one foot in front of the other, your robot will realistically never walk, because you need to account for the dynamics of the movement, the inaccuracies of the sensors, the friction with the ground, and potential perturbations from the outside, like being pushed. So training new policies is definitely doable, but you can't train a policy to get a genuine long-term companion in your home (well, technically you could, but that's a whole other range of policy: a research-grade problem, and the resulting neural network would probably be huge and never able to run on such a small compute). Instead, you need a few more building blocks: voice recognition, facial detection/recognition, localization, some simple navigation, and a behavior layer, which is less technical and harder to describe. As a rule of thumb, a prototype of any of these is within reach of anyone if they have a modern coding agent and if the computation doesn't run on the robot (meaning the robot can reach the internet or an external computer and use external APIs). A rough prototype for each of the above can be a chill weekend project. As always, going from the prototype to a high-quality implementation is another story, but still doable for most if you put in the time and test a lot. However, having all of this run locally, on the onboard chip, I would call a very hard task. You'd need some very clever engineering to get a good enough version of everything working together. But I can't say it's impossible, because I've seen some crazy efficient implementations of all of the above. And here we don't need a super high success rate: it's okay if the duck sometimes does something a bit random. So I wouldn't be surprised if folks managed a fully local, cool, simple version of this autonomous companion. For example, you don't need Whisper-grade open-vocabulary transcription: being able to recognize a few keywords would already be great for interaction. One of the fastest ways to turn this robot into an autonomous companion would be to stream Microduck's sensors (the camera, the depth sensor, the IMU and the joint positions) to a computer, and use a software stack like the one we use for the conversation app on Reachy Mini: a local VAD, then a cloud speech-to-text model, then a call to a frontier LLM that decides on actions. Actions could be quacking, looking at things/faces/a direction, walking to a given position, sitting down, playing an emotion, kicking or doing a somersault. A SLAM algorithm would run in parallel so the robot can localize itself in your home, and you would prompt the LLM for a lifelike behavior: moving to random places, sometimes following people, and having emotional responses that fit what you're saying and the duck's environment. Of course someone is going to build a Tamagotchi-like layer on top of this, and it will be insane. How much demand is there for a software suite that would do all of this? I for one think it would be super cool. A word about the humans. We had five engineers backstage, and they are pretty much the core team behind Microduck. Their impact per head is absolutely off the charts. Antoine Pirrone piloted Graphite, Augustin Crampette Lavender (the rebel one), Matthieu Lapeyre Sky, Coralie Deplane Cream, and Pierre Rouanet was stressing over the Wi-Fi. Give them a follow! (Antoine @antoinepirrone, Augustin on LinkedIn, Matthieu @matth_lapeyre, Coralie @Drenaya_, Pierre on LinkedIn). I get visibility because I communicate a lot, but I mainly work on Reachy Mini (you might want to check it out: it's a very cool robot, and it ships much faster than Microduck :) ). My contribution to Microduck has been minimal: I'm mostly a first power user. My main added value at GTC was having hands big enough to carry four Microducks below the stage. Where I take pride is that Antoine and Coralie are former students and former interns of mine. I'm therefore entitled to 3% credit on anything (good) they do, for life. If they do something bad, however, it was clearly the fault of their parents. About the super smart engineering that can be done on board: Antoine just released two very cool blog posts (links below). The first is on odometry, the robot estimating its own movement from its own internal sensors. The second is about SLAM, which allows long-term localization: it adds the depth sensor to the odometry to get a stable position over time. Think of it as recognizing known landmarks in your home to know where the robot is. The crazy part is that all of this runs on the robot. So this gives me hope that we'll be able to do a lot of cool stuff on board.