Google's Project Astra Is an AI Assistant That Can Respond to What It Sees

At I/O 2024, Google made lots of exciting AI announcements—but one that has everyone talking is Project Astra. Essentially, Project Astra is what Google is calling an "advanced seeing and talking responsive agent." This means that a future Google AI will be able to get context from what's around you and you can ask a question and get a response in real time. It's almost like an amped-up version of Google Lens.

Project Astra is being developed by Google's DeepMind team, which is on a mission to build AI that can responsibly benefit humanity; this project is just one of the ways it's doing so. Google says that Project Astra is built upon its Gemini 1.5 Pro, which has gained improvements in areas such as translation, coding, reasoning and more. As part of this project, Google says they've developed prototype AI agents that can process information even faster by continuously encoding video frames and combining video and speech input into a timeline of events. The company is also using their speech models to enhance how its AI agents sound, for a wider range of intonations.

Google released a two-part demo video to show off how Project Astra works. The first half of the video shows Project Astra running on a Google Pixel phone; the latter half shows the new AI running on a prototype glasses device.

You May Also Like

In the demo video, we can see the user using their Pixel phone with a camera viewfinder open and moving their device around the room while asking the next-generation Gemini AI assistant, "Tell me when you see something that makes sound" and the AI responding by pointing out the speaker on the desk. Other examples in the video include asking what a part of the code on a computer screen does, what city neighborhood they're currently in and coming up with a band name for a dog and its toy tiger.

While it will be a long time before we see this next-generation AI from Project Astra coming to our daily lives, it's still quite cool to see what the future holds.