Pepijn Kooijmans from Hugging Face's LeRobot team discusses how AI and open-source technology are democratizing robotics, making it accessible to anyone. He explores recent breakthroughs in robot learning, safety considerations, and the vision of building robots as easily as assembling LEGO, while highlighting creative community innovations.
▸ Read the transcript
What's one random robot prototype artifact that you have seen recently? For example with 3D printing, right. What we did with the SO-100, like making robots as What we did with the SO-100, like making robots as cheap as possible and as accessible as possible If more people are gonna build with this technology, you get a much more fun future, with different kinds of robots that look different and do different things, After a couple years I kind of saw that the field which I really loved which was robotics and mechatronics was kind of slowly and mechatronics was kind of slowly starting to merge with AI And it's a very open I would say part of Hugging Face where everyone basically with a good idea can try it out, can pitch it and can build something and there we try to which makes it very exciting to see what's going to stick and what's not and why so there's a lot of learning to be done Everyone can build their own kind of robot or version of a robot And that's also what we want to enable everyone to be able to do. Hello guys, here we are in TechEx with Pepijn Kooijmans from Hugging Face. Hello Pepijn. Hi, nice to meet you. Nice to meet you Pepijn. So we're gonna go a bit into your career and how did you get into leading the robot at Hugging Face as well? Yeah. So first of all I wanted to ask you how did you get into autonomous robots and then how did you get into LeRobot at Hugging Face? Yeah, I mean I'm not sure how far back you want to go but of course as a kid I was always excited about technology, robots and technology that had moving parts, etc. And that's why I initially studied mechatronics as well. So a combination of hardware, software and the interaction between those systems. After that I did a bit more of a software job with a bit of computer vision and AI at another scale up. And after a couple years I kind of saw that the field which I really loved which was robotics and mechatronics was kind of slowly starting to merge with AI back in I guess early 2021 with you know all work coming out of Google Everyday Robots project, etc. And seeing all that research made me very excited to you know combine these two fields which I was very excited about and start working in that field more and more. And so yeah you know after again going very deep into that I trained one of my first like end-to-end robotics models, I experimented with building robots myself and then eventually Hugging Face, which had just started LeRobot as well, kind of came onto my path. We were doing very similar things. I was also building robots you know applying this technology LeRobot wanted to do this for a broader audience and that's how I kind of ended up there and that's it. So you were more let's say being a creative in the field which was a niche field and that's why it was easy for you to access Hugging Face and the proposal for open-source robotics, because you also represented what they wanted to do. Yeah so I think a couple months earlier, Remi originally started this project. Remi Cadene had previously been at Tesla, working on the Optimus robot. Now he does his own startup so he left a bit more than a year ago. So he originally started LeRobot. He pitched it to the team at Hugging Face, and that's when a couple of months later, I joined, and that's how we kind of grew the ecosystem from that point onwards. I was wondering: Hugging Face is an open- source company. How does it operate from a company perspective? From a company's perspective you of course have a lot of people working on the hub and the project itself. So we do a lot of storage now. We do, of course you need a lot of networking, you need the whole platform for people to upload datasets sets and models and we mostly work now on the open source and science team, which is a bit different because we of course leverage all the technologies that the hub team basically has built but on top of that we create libraries or tools or side projects and that's a or tools or science projects and that's a very open I would say kind of part of Hugging Face where everyone basically with a good idea can try it out, can pitch it and can build something and there we try to really maximize impact. So what we also try to do with LeRobot is just have as much impact as possible in the field and enable as many people as possible to become AI builders to use this technology for themselves and that's also what we as a team are very focused on within the open source and the science etc. Your team particularly in LeRobot, if you would have to define the whole division how would you define it? I think we have a team of a lot of very smart people, very passionate about robotics and robot learning and I think if everyone has the same passion it's quite clear for everyone what to build or what to do next and it's just a matter of keeping discussions about where we think the field is going and just building as much as possible to get that feedback feedback from the people using the library and the ecosystem basically. So basically, everyone on the team, myself included were mostly throughout the day just trying to build things that hopefully the community actually uses and that's very exciting that like everyone actually builds every day. I was wondering: before, we used to have robots that learn with computer vision and other techniques with a lot of data Are new techniques popping up for robot learning in 2026? Yes for sure I think this year has been the most exciting year and every day there's new ideas that are being tried in the research community and in the industry. Some parts come from robotics, from the robotics field historically, and some parts come from LLMs and from other AI-modelling techniques so it's very exciting because robotics has a lot of different parts. What we see now is of course you have video models, you have at the same time LLMs also being trained on robotics you have more of this in-context learning which means that we can maybe just prompt the models with a video of a human doing a task which is very exciting and then of course on the navigation side of robotics there's also a lot of exciting stuff going on so because the ecosystem has grown so much there's so many different directions that are being tried out which makes it very exciting to see what's going to stick and what's not and why so there's a lot of learning to be done there and it's exciting to read, also a bit overwhelming I guess. A bit overwhelming because there's just so many people suddenly. Historically, robotics has been, I guess, maybe a bit more of a smaller field, especially like the people that believed in end-to-end learning within robotics and now it's suddenly become so big that it's maybe a bit overwhelming as well because so many people are now thinking about it which is great but it also generates some noise if you want to just learn about the field. I was also wondering whether, for these kinds of training or these kinds of robot actions, any kind of physical or physics constraints are necessary for robots or is it just training, like machine learning and AI training? I think for robotics it is important to know about how a robot works so how the controller works, because it's still being used by the AI often and then it depends a bit on which part of robotics you're talking about. For locomotion, so for walking around and learning whole- body, like lower-level movements, it's very important that you correctly have all the physics of the robots in the simulation for example for the very heavy training stuff of training manipulation of picking, placing etc. It's more about making sure that the data is just very good and that there are no issues, for example with how you measure it or how you record it But I would say it's maybe less about understanding the physics is less important there because it's already grasped in the data basically So you don't need to be very... Yeah, it's different than locomotion in that way. The field is becoming massive with open source and everything. How is the industry or the regulators handling safety, security regarding robots? I mean, historically, of course, luckily, there is already a good framework about safety and robotics It is only that now the brain of these robots work differently. and of course you have this whole humanoid market for example coming as well So I think there's so much new research coming in, and this new trend will need a bit of a change there in regulation or on that side. So we need to think about how to structure that because it's new. It's still really new, so there are a lot of proposals coming in. and at the same time of course because there are so many directions still being explored and we're still figuring out what works and like, slowly now... Well, not slowly, but now only since this year, I feel like people are actually going to deploy these robots. It's also, yeah we're still early so it's hard to actually say that this is going to be the way that it's going to unfold for the next five years So I think, because we're early, it's also important to first learn about how the system works and how they will work before committing too early to someone saying this is how it's going to look like Because, as a field, I don't think we're there yet. We don't know exactly how it's going to look yet. We don't know what it's going to look like, but if I paint you this picture, Now people are vibe coding software applications. who will vibe code, on one side, the software frameworks for robots and also build our own robots as if they were LEGO? Imagine: as if they were LEGO? Imagine: you basically code basically the whole robot. Do you see that coming in the next few years? Yeah, for sure. I think parts of designing a robot are already becoming simpler. We're definitely not there yet, but I think the trend is very clear. Like, engineering is just going to become maybe engineering is just going to become maybe not easier, but more accessible in a way. and also all these systems will allow us to iterate faster. So I think, yeah, that will definitely come which is very exciting, right? Because then maybe everyone can build their own kind maybe everyone can build their own kind of robot or version of a robot And that's also what we want to enable everyone to do, like help everyone understand AI, but also help everyone understand AI, but also build in this space. also especially with robotics because I I always say there are so many applications for robotics so I think the the most exciting thing is to have everyone build robotics and build robots and use this technology because everyone has, you know, their problems maybe that they want to automate Everyone has a different view and a different industry. So I think it will be very hard, you know, for a single person to do all of this, to automate every single thing, because I don't even know what happens inside every little factory So I think it's very important that everyone is actually enabled to build robotics with this new technology. Following up on this question of accessibility, I was wondering if you think also third world countries are going to be able to create their own robots in the future? Yeah, I hope so. I mean, also, for example with 3D printing right what we we did with the SO-100, like making robots as cheap as possible and as accessible as possible, so people can learn the technology and develop new things on top of it. I think develop new things on top of it. I think that's one of the most important goals that we have, because it is impactful. We also saw a lot of countries from all over the world for example last year in our robot hackathon that we did So yes, I think we really want to enable everyone to build. And what's one really weird robot or unusual case that you've seen? What's one random robot prototype artifact that you have seen recently? Many. I mean, I've seen a dentist robot picking up tools, or a dataset of it. We had this very cool community member who built basically a string robot with cables, that could fly across the whole room to pick up stuff. Yeah, the community is much more creative than I am by myself. We're in the office, we have these simple tasks, but yeah, not as creative as the community is. They're way more creative, which also shows that if more people shows that if more people build with this technology, you get a much more fun future, with different kinds of robots that look different and do different things, which I think is great And, yeah, a few last questions. If you had your younger self here, starting as an engineer, not knowing about AI yet, what would you tell him? Just be excited, and don't be afraid, I think, to start building something. Imagine what you might want to see for yourself, or what you would want to build for yourself, and don't be afraid to just get started. Order some parts, maybe, or a robot. Especially now with AI, I think there are so many things that you can just figure out. But you just have to have a nice side project for yourself and just start doing things and building things. You started because you had a vision of what you wanted to create, or you just wanted to explore your creativity and create things? I think a bit of both. Yeah, I think especially once you see what robotics can potentially do now, yeah, it's very exciting. So you did it just for the sake of doing it, right? Not for any other ulterior goal, like getting a job or... No, just because I like the technology, and, yeah. And last question: if you could connect with somebody, either in the past or now, who would you want to meet and connect with? That's a very good question. I probably need to think a bit more deeply about it. I might just say, you know, maybe the real-life Tony Stark from the movies, because he's also a builder, right? So I think that would be cool. Do you think we'll approach the future like Iron Man, building robots with AR, augmented reality? I mean, we're very close to it. I feel like you can just speak to a computer and it can already build some things. Yeah, I really feel that it's becoming magical, the way you can directly talk to an LLM and have it do things and actually implement your creativity or your things. What are we lacking to make the picture come faster? More compute or more expertise in combining and integrating all the technologies? Because the technologies are there, right? Yeah To have this kind of Tony Stark... Yeah, probably both, and a little bit of patience. But, you know, the field is moving so fast that we'll get there very soon, I think. Perfect. Well, that was it, Pepijn. Nice to meet you. Thanks so much.