Home /Nectora Videos

Nectora Videos

I Thought CEOs Were Useless": The Google Scientist Who Raised $100M for Voice AI | Neil | Gradium

Conference Amsterdam Interview
Watch YouTube LinkedIn Soon

Neil Zeghidour went from AI researcher to Founder & CEO of Gradium, raising $100M to build the model layer for voice AI. In this conversation recorded at HumanX Europe, Neil takes us through his journey from Facebook AI Research and Google Brain to Kyutai and Gradium — and explains how research around SoundStream and AudioLM helped reshape generative audio. We discuss why small teams can outperform Big Tech in voice AI, how four people built Moshi in six months, Gradium's 50ms text-to-speech latency, the origins of modern audio language models, and why real-time translation could be a matter of months rather than years. Neil also talks about making the transition from scientist to CEO after once believing that “CEOs were completely useless” — and how he now applies scientific experimentation to company building and go-to-market. Highlights 00:00 — “CEOs were completely useless” 00:37 — What Gradium is building 01:28 — SoundStream and the origins of neural audio compression 03:22 — Applying the LLM approach to audio 03:47 — “In one week, we invented voice cloning” 04:32 — AudioLM and the early connection to Suno 04:56 — Leaving Google and moving toward real-time AI 05:57 — Moshi and

What we learned

  1. 1 Start from the most important problem you want to solve and then think about the method, never think about the method first and then say what application it can have.
  2. 2 You can bring scientific methodology and experimental discipline to everything, including go-to-market strategy, by creating experimental methodology, having the right APIs for measurements, and building self-improvement loops.

About the guest

N

Neil Zeghidour

CEO · Gradium

Gradium is a frontier AI lab and company that trains voice models to enable real-time voice interactions with LLMs, creating models for speech-to-text transcription, text-to-speech synthesis, and voice agents.

Challenge
Making people actually use models in the real world and pay for them is much more difficult than making them download a free model, and you need a certain level of reliability for complex industries like medical appointments handling severe pathologies.
Lessons
Start from the most important problem you want to solve and then think about the method, never think about the method first and then say what application it can have. You can bring scientific methodology and experimental discipline to everything, including go-to-market strategy, by creating experimental methodology, having the right APIs for measurements, and building self-improvement loops.

Recorded at

Amsterdam →
▸ Read the transcript

nontechnical people and CEOs [music] in particular are completely useless you know and that was not the right way to approach research and then I I changed my mindset I had never planned to be an entrepreneur at the same time I had never planned to be in [music] tech startups are really doing a fantastic job because again there is no real advantage of being a super big company I developed it mostly for Google [music] meet so that you will have good audio quality even when you had bad uh network and I think what changed really afterwards [music] is when I left Google to create I started looking into real-time interaction so not offline generation realtime interaction. Hello Neil, nice to meet you. How are you? Fine. So please tell us who is Neil. So I'm a founder and CEO of Gradium. Uh Gradium is a frontier AI lab and company around voice. So we train voice models to turn any interaction you would have with a with an LLM from text to voice. So we create models for transcribing human speech into text in real time. Synthesizing machine text into natural speech in real time as well. So that you can create voice agents that are very fast, accurate and sound natural. And you started a while ago some years ago in in Google with models like soundstream and audiom and soundtorm. there was a natural evolution towards your your career. Could you explain us a bit about your origin and how did you start it? Yeah, absolutely. So, it's an interesting story because it has been what led us uh to create gradient basically. So, in 2021, I had been working already on voice and audio for a while, but I wanted I looked at the very specific problem which was audio compression and so I wanted to create a codec like MP3 or Opus, you know. So a format of compression of audio and the way MP3 opus work is what we call lucy compression. So you lose some information but it's made in a way that you lose information that is not very relevant for the human ear. So you can compress very heavily and not degrade the quality too much, right? And so instead of doing that with all this expert knowledge of human earing and so on, I just wanted to learn a codec end to end. And the soundstream was the idea of training a neural network to compress audio and then reconstruct it at high quality. And I developed it mostly for Google meet so that you will have uh good audio quality even when you had a bad uh network, right? So it was really not with generative AI um in mind but then interesting what happened after is um I got interested in generating audio and back then people were using generative adversal networks but that worked well for like I don't know 5 seconds audio very basic things uh but it could not model complex structure like a long sequence where you're going to have multiple musical instruments or you will have multiple speakers you know all the things that made make an actual audio in real uh reach it could make like the voice of Google maps you know or something like that in batches right in batches you you have to use it in yes exactly and it could not model very complex things right why was that why couldn't because that was linked to the fundamental limitation of the uh architecture which was a convolutional neural network on top of of the waveform so what we did is we took inspiration from LLM so the way LLM what they what they do is they represent text as token hands and then they train to predict the next token on very big corpora right and so what we thought is okay our codec it's kind of a audio token so we can take audio we compress it and we train a audio language model to predict the next audio token and then we can reconstruct audio with the codec we train that and in one week we invented uh voice cloning music generation uh like it just work straight away so it was a big landslide inside Google because so like internally it it replaced quite quickly all the stuff that were uh being used a bit like LLM for text. Before that audio was very scattered, right? Like the models for text to speech were very different from the models that the Magenta team for example was using to do music that were very different from the ones that were doing speech to text. And now for the first time you could unify everything into a single framework and so we could ship speech translation, TTS, speech to text, you name it. So it was really noise removal as well, right? Yeah, you could do pretty much everything you wanted. We made a paper. We got uh actually one of the first byproduct of that was uh so the music generation startup started as a open source reimplementation of audioM and uh and you know like this framework was quite successful but it was also one of the two main frameworks. The other one was diffusion. So people were either using audio language models like us or they will use diffusion models. And I think what changed really afterwards is when I left Google to create I started looking into real-time interaction. So not just offline generation, real time interaction. And when you do real time audio language model makes much more sense than diffusion because it's auto reagive. So it can you can start listening as it generates the sequence, right? You don't need to have it generate the whole thing. So it was like such a good fit for so before it was sequential prediction and it needed the whole sequence to predict the next sequence. It's more than with diffusion you generate the whole sequence at once and then you can start listening. With audio you listen as the sequence is generated so you can start listening right away. And so basically there were like these two families of models diffusion and autography models kind of fighting one another. But when voice got more about real time interaction rather than offline content autogressive models just wiped out diffusion. And if you look today any model in the world almost is auto reggressive audio language model. So um and that's what has led to gradient because we when we did QI which is a nonprofit lab um we did open research we made the first real time conversational model called mushi which was full duplex so now people know the term full duplex I did not even know that back then the fact that you know it listens and speaks at the same time like you would have with a human we open source that and it became the backbone for quentts minimax models from meta model from sesame models from mistral and so the idea of QAI arise who proposed it how do you create a team of uh researchers yeah so it's interesting so u basically before cutai and gradium I was first at Facebook a research when yanlukan opens the paris office and then I joined Google brain which was then merged into Google deep mind in both cases I was recruited as a scientist and my job was to create uh knowledge you know and discoveries and share them with the world through scientific papers and sometimes open source releases and the relation of big companies to uh open source changed in particular after GPT as it was becoming more and more useful to the business and and important strategically you know generative AI it was less clear whether we wanted to open source and publish our discoveries and for me it was very important to recreate an environment centered around sharing knowledge and so basically the idea of QA is that we didn't to go into academia because we needed money to have GPUs. Uh we didn't want to do a startup because we didn't want to have a product. We just wanted to do research and we wanted to own ourselves the decision of open sourcing and releasing what we did. And so QAI was like the only thing that checked all these boxes because it was a nonprofit lab with very large funding. So we got 300 million euros of donations in 2023. So we could get like a 1,00 uh H100 GPUs, total freedom, no strings attached and just open research and and the outcomes of that were such that we got more and more inbound requests for making commercial versions of our models and we were seeing voice skyrocketing as a category, right? And so we decided to create video. Did you use the open source model to further develop this these models at the end in QI? Uh like what kind of open source models? So did you develop a open source strategy so people could give you feedback of your Oh yeah, I think we could have done a better job back then to uh engage more the community with us. Um but now I think for example a good example of that is the open source ondevice text to speech of QI pockets where uh now everybody can train it in their language and and and add it as a contribution to the model zoo of of pockets. So I think that's that's a good example. So how did you started the gradium and how was the transition towards a researcher and probably managing research teams towards a CEO that now needs to pitch funding more engineering maybe I had never planned to be an entrepreneur at the same time I had never planned to be in tech you know uh the thing for example I just wanted to do a PhD in computational neuroscience and uh I get the opportunity for Facebook to fund my PhD and then I will work at Facebook I'm like yeah sure you know let's do that So um so it kind of came to us because again we got so many requests for making commercial version of our models that will be much more robust than the open source ones um that it was out outside of the scope of a nonprofit to do that right and at first we just refused and we said you know we don't do that and so on. Then it was so weird because like it like the opportunity to make people use our models at gigantic scales and being the main models that people use in their daily interaction. we were kind of refusing to seize it and then at some point it was enough and like no we have to you know we have to take this opportunity it's impossible to miss this impact making papers and open source is great but then having people actually using your models in the real world it's much more difficult because making people pay for your models is much more difficult than making them download a free model you know and you just want to be good enough that you're really trusted by complex industries you know and uh very high standards of customers like you know we have customers handling uh medical appointments about sometimes severe pathologies with our models you know you need a certain level of reliability for that so we did the fundraising we raised $100 million in seed um and I transitioned from scientist to CEO uh I really enjoy the transition and uh and I think it's easier now than ever because back then before coding agents Um becoming CEO meant giving up entirely on the technical work right because you could never get enough time to focus and uh otherwise you will do a poor job as a CEO right so you had to give up on that but now with coding agents I I can make them and they're working you know in the background and I can obviously I try not to be on the critical path of our most important technical projects but there are some small projects here and there I can can grab to evaluate a specific thing or to tweak our synthetic data in one direction or another and I do that with my agents as synchronously and that works quite well. So now I think it's a great it's really a great time for technical founders to transition to CEO while still staying technically uh more involved. There are a lot of labs in these big companies Google, Facebook etc. How do you differentiate yourself as a small startup uh with them and what's the strategy of gradient to compete? Yeah. So I think one thing that is interesting in voice is unlike text models that require from day one a gigantic infrastructure for training like a two two trillion parameter model u like it's very hard for a challenger to emerge right because you need very large capital you need very large infrastructure and all the connections and so because now even today if you had a lot of if you had 1 billion to buy GPUs you cannot find those GPUs so you need to have the connections the network like the influence to make it happen right in voice is very different. Our models are trained on a few dozen GPUs. They are very small and so um it's much more about the deep expertise of a few people rather than having a very large team and very large infrastructure. So to give you an example when we did Moshi it was the first realtime conversational model. We trained it from scratch. We trained the text LM from scratch and the voice. We invented this uh full duplex algorithm on top of it and we trained the codec and so on in six months with four main technical contributors and we are the first in the world. And then we did real time speech to speech translation on device. We are the first in the world with two two people two people and a half. Um so every time we've been doing something we are the first in the world to do it. And so now we are bringing more and more talented people to the team. But voice you can see it. It's really for me the field in AI where u startups are really doing a fantastic job. Like you you look at at benchmarks startups are are the top um and because again there is no real advantage of being a super big company to do voice uh except for distribution but not for training the models. Um and there is a big advantage of being a small team that can go very fast. And basically what is the like the the vision I have for Gradium is we really focus on the model layer. So we want to be the the preferred model provider for everyone building applications around voice right. So companies making voice AI customer support personal assistants AI therapist AI NPCs you name it. We want to make, you know, get their feedback, make models that they feel comfortable building with and serve these models uh for a good price with extremely good latency. We released yesterday the fastest text to speech model in the world, 50 mcond latency. So, you know, that's who we want to be. We really want to be just the model layer and power all these applications. How close are we to a world where there is live streams on Twitch happening that have real-time translation meetings where there is also real conversation and that's one of the things that passionate me the most and the example of Twitch is one that I'm very interested about because it needs both to be very fast and very accurate and fun right you you the voice needs to be lively and uh if you have a robotic flat voice for the translation people are not going to like it right And uh so if you can make realtime translation and that people find it entertaining, I think that's a fantastic uh uh thing to do and I would say it's a matter of months rather than a matter of years. What would you have tell the young Neil that started in Facebook in his early days is you have it in front of you. I think one one thing that was very successful for me u and that changed a bit after my PhD is when I started research I wanted to um like find a smart solution to problems that were not very important just to show that I was smarter and that was not the right way to approach research and then I I changed my mindset and I didn't really care about the method or the algorithm I was just looking at a problem I want to solve and one of my mentor David Gon who worked with me at Google is the one who brought me to that when we started working together and he was a legendary researcher you know I used to only think about the algorithm and he told me you know let's focus on one problem you want to solve and he told me I like the subtitles on Google to have like dashes for each speaker and then you know you start from this objective right like that you want to achieve and then you you go to okay what like are the current algorithms good enough no we need to invent something Okay, let's start inventing a new algorithm so and always starting from the most important important problem to solve and then thinking about the method and never think about the method first and then say okay like what application can it have? I think that's the most important thing. Any lesson for researchers that want to transition towards a more commercial or operational or business role? Yeah, also it's one thing I learned along the years is u honestly I used to think that I I'm very honest. I used to think that nontechnical people and CEOs in particular were completely useless. You know like that's the only thing that matters was uh technical work. Now I realize um you can be you can try to be as creative and have a taste for experimentation in your job as CEO that you do as a researcher. So you know like the way we approach go to market there are a lot of experiments we are doing on SEO on GU how can we optimize the referencing by LLN and so on and you can create experimental methodology have the right APIs to do measurements create self-improvement loops and so on and I'm sure this discipline and this scientific methodology can be brought in pretty much everything. Thank you so much. Thanks.