Digital AI and Physical AI

With one of our research, we are moving further into the 3D World of things - Physical Intelligence and Robotics:

Would it not be amazing if we are able to replicate our experience in the world of robotics and AI?

Understanding how we can build and replicate what we are experiencing and teaching it a robot to help us along the way?

Not only for the understanding and development of robots, but also for advancing our own understanding of the world and how we operate in it.

Learning our worlds

We do many of our physical experiences without thinking about them. We wake up in the morning, go to the bathroom, make ourselves a coffee, go to work. We kiss each other goodbye and we take a bus or subway, bike or our our feet to bring us to the place we work at.

There we move across rooms, we walk, we talk, we are active physically. We move our mouths and speak, we hear, we listen and we engage.

For us, these tasks are normal, they are like breathing. But everyone who ever had an injury, lost a limb, or had another experience of losing part of that normality, knows, that while we take these things for granted, they are not. And learning them back is a painful, time-consuming process that takes effort and time and makes us aware of how complicated each individual step is to re-learn and to understand.

How research allows us to understand these processes

In our research we came across various companies in the space and specifically loved Physical Intelligence. They recently raised a $600m funding round and work on building precise autonomous robot arms that are trained to work on various tasks in the physical world - including folding clothes, putting boxes together and becoming a professional barista.

They use a "two-stage" training strategy to build a generalist robot brain, including of a pre-training phase and post-training/IRL phase.

The pre-training, generalist brain is a vision-language model (VLA) trained on a massive mix of internet data to learn common sense and cross-embodiment data, which are demonstrations from many different types of robots using Flow Matching, which allows the AI to predict smooth, continuous physical movements rather than jerky, robotic ones.

The post-training/IRL mastery is then setup using a method called RECAP, where the robot "practices" tasks autonomously; if it fails, it uses Reinforcement Learning and human corrections to understand why it made a mistake and how to recover, essentially teaching the robot to learn from its own experience.

Our Own Research: 3D Worlds and Autonomous Character Interactions

Our own research focuses not on robotics but on 3D Worlds where autonomous characters interact with each other and collaborate and play scenarios and let stories unfold in that digital space.

For our research, we came across an interesting perspective - what if we do not only build a digital world that is basic in its activities, but what if we give the humans and characters in our world specific arms and robotics that we can train to do actions - ways to later be transported into the real world and trained on real-world robotics arms - how to build that replica of a world and how to simulate how these characters are working and interacting with each other.

It is still some stretch towards that merging of intelligences, if digital and physical, but if you think how the robots learn, then it is not that different to how we learn and the real world becomes closer to the digital world, while the digital becomes more engrained in the physical world.