Welcome to another installment of Coexist, a space dedicated to reflecting upon and interpreting the intersections between design, technology, humans, humanity, ethics, and justice.
If you haven't yet subscribed, I invite you to join us below.
Hello, dear readers,
In the last chapter, we explored our relationship with machines, with a special focus on AI, its evolution, and its imminent impact on our lives.
We delved into the concepts of generative AI, LLMs, and neural networks, examining how they work and why they have been so successful in the last year or so.
In this chapter, I want to explore our current relationship with robots, how we interact with them, the main challenges of the industry, and how design is and should be implemented in this growing discipline.
The design of our relationship with machines is nothing new, but we are living in a pivotal moment in our history as humanity. We are developing agents that are capable of learning, creating and making decisions by themselves. Their interpretation of our world will be unique, no other technology in the past has been able to do what is possible now. This could have significant implications for our future, as things could escape our control without us even noticing. This is why designing how these agents communicate with us is extremely important. We will need to understand their processes and intentions and ensure we can intervene when necessary. The writer of Sapiens and Homoedeus Yuval Noah Harari, who has been very critical of the development of AI and the need to control its development, thinks that this technology could have the impact to even change our history, our culture and our view of our world.
Now, how is our current relationship with these entities? What concepts and disciplines are important to understand when we think about our interaction with them? What is their status today, and how the new advances in technology are shaping the future of robotics?
There are complete interdisciplines that focus only on how our interaction with robotics is going to be. Concepts like Human- Robot Interaction (HRI), Social Robotics and Collaborative Robotics are key to understanding where we are today and where we are headed. Now, I will analyze these different interdisciplines in the field of robotics with the idea of exploring the importance of having transdisciplinary teams in the development process of these new technologies.
Human - Robot Interaction (HRI):
As designers, we are familiar with disciplines like Human-Machine Interaction (HMI), Human-Computer Interaction (HCI), User Interaction (UI), and User Experience (UX). They are related but have different focuses. HMI is a broader term that includes robots but is not limited to them. It involves the design and implementation of interfaces that facilitate communication and control between humans and machines. UI and UX are often linked together and are key disciplines in the development of digital products. From my personal experience, HMI is a term most commonly used when designing the interface of physical products.
HRI refers to the study and design of robots that can interact with humans. This interdisciplinary field encompasses aspects of robotics, artificial intelligence, psychology, cognitive science, and human-computer interaction. The goal of HRI is to create robots that can effectively and safely interact with humans in all possible contexts, from our homes to space exploration.
The origin of HRI can be traced back to the early days of robotics when researchers started exploring the possibilities of creating robots that could collaborate with humans. With new advances in technology, the boom of Large Language Models (LLMs) like ChatGPT, and the increasing presence of robotic products in the market, the importance of creating machines that can interact with humans in a natural and intuitive manner has become clear. This involves creating user-friendly control systems, feedback mechanisms, and communication methods that enable seamless interaction. UI and UX principles are also applied to enhance the overall user experience when interacting with robots, ensuring that the interaction is not only functional but also enjoyable and satisfying for the user.
So, if we analyze the incredible success case of OpenAI with ChatGPT-3, which launched around 2020 and went viral in 2022, we see that its success was due to its user-friendly intuitive interface. ChatGPT was not the first LLM created, nor the first in the market, but its 3.5 version was for sure the more intuitive. The way users could naturally communicate with a computer, as if they were talking with another person, and receive clear, natural answers was the main reason for its success.
With ChatGPT, we mainly communicate via a chat in our web browser or app, in this case we talk about Human-Computer Interaction (HCI). However, once the same model is integrated into a robot that will “talk” with users and interact in the physical world (beyond voice commands), we are dealing with HRI. In this case, the development team (engineers, designers, psychologists, etc) will need to think about the physical actions of the machine, its tone of voice, and how it moves its hands, face and posture.
In their YouTube channel, the startup Figure published an update video in March 2024 showing their humanoid robot Figure 01, connected with OpenAI’s pretrained technology. (Open AI is one of the main shareholders of Figure). We can see here the amazing result when AI and robotics are integrated in the same product. This will be the future of smarter and more autonomous robots that will understand their context and perform almost any task effectively.
The advances in generative AI, and their integration with robotics are game-changers, because it is solving one of the big challenges with social robotics. These systems will be able to understand the natural language and communicate more transparently and clearly with us. Not to mention how it simplifies the programming and development process, as roboticists will not need to manually program each robot behavior. These robots will have the capacity to adapt and learn from experience to improve their actions. I will return to this point shortly.
Collaborative Robotics
Before deepening on the HRI specifics, I need to talk about Collaborative robotics, meaning when robots are designed to work with humans safely and effectively in a shared space. The key and biggest challenge for developers and designers here is communication, specifically bidirectional communication. The human that is working next to the robot needs to understand what it is doing and what it will do next. Conversely, the robot must understand the human’s actions and predict their next moves to quickly adapt its actions.
Safety of course plays a crucial role in these interactions. Robots that move in the same spaces as humans need to understand when to stop their actions, adjust their forces, and change their movements and speed depending on the human’s location and future movements.
There are many ways we humans can communicate with robots. We can put sensors on us to indicate the robot where we are, we can use our voice and/or our movements to indicate our position and intentions. Depending on the design and function of the robot, different strategies can be used to predict and understand human actions. These strategies can involve cameras, radars, force and temperature sensors, microphones or a combination of all.
The most important goal is to ensure a bidirectional transparency, where both human and robot quickly and clearly understand each other’s actions. This process needs to be fast and straightforward. Transparency in HRI is a challenging but essential topic. The idea is that every action the robot is doing or going to do needs to be clear and transparent for us. The robot must communicate its status, whether it is on standby, active or off, and if it is looking or sensing us, if it is about to move, jump or run from one second to another, the possibilities are infinite and it is easy to imagine the big impact of design choices on how the robot communicates in each situation.
Latest developments in generative AI (Check our previous chapter about Generative AI, LLMs and more - Chapter 4.1) and machine learning are greatly impacting HRI, as robots are becoming better at predicting our movements, understanding our requests, and learning from observation. One approach in AI that is going to have a huge impact is Vision-Language-Action (VLA) models.
VLA models translate visual and language inputs into motor actions. It works by integrating:
Vision processing: It analyze how the robot “sees” and understand the content of the scene, the input could be different, from videos, images or the 3D shapes of the surfaces it scanned, but the breakthrough is not only the input of the scene, but the understanding of what is “looking” and how this context change every second.
Language processing: it allows the robot to understand natural language instructions, understand the human intent and generate descriptive text of the visual content or intended actions.
Action control: The model is going to be able to generate or predict actions by combining its understanding of the visual input and language instructions. This is a very important change in the game, developers do not need to “program” every single action of the robot, but it will automatically perform any task, that can be a very simple command like picking an object to prepare a complete meal.
We can imagine how these types of models are going to have a big impact not only on the development, programming and design of robots, but on our interaction with them.
At this point, we have experienced how easy and fluid our conversations with LLM models like ChatGPT have become. Now, having this technology embedded in a robot that will be able to understand your natural language and communicate with you in a fluid manner will make the interaction much more intuitive, efficient and accessible to a wider range of users. Users will not need to be technology experts to interact with robots, they just need to talk, or even show the robot how to do a task.
Also, robots will be able to understand their context as never before. They will be able to adapt to any change, plan and execute complex tasks more accurately, and safely around humans. VLA models are still in development, it is likely some autonomous robots that are already out in our world are using some elements of these models, but are not publicly disclosed by companies. As I mentioned before, recently we saw how Figure is integrating a version of LLM from open AI in their humanoid robot Figure-01. The demos look incredible; the robot is able to understand natural commands, analyze and understand its context and plan and perform the needed actions to complete the task at hand. I think it is safe to say they are using their own version of VLA.
In the last months when this article was written, the big hype in robotics has been humanoid robots. It seems many companies are competing in the race to create the first multipurpose humanoid to hit the market. It makes sense that models like VLA will impact humanoid robots in particular more than other robot architectures. The main idea of creating a humanoid robot, is to have a robot that is able to navigate in a world designed for humans. If we want robots that are able to navigate our world, it is necessary that they can do any type of task, from household chores to moving objects in a factory or warehouse, to assembling other components, to taking our dogs for a walk, and most importantly, that can understand and interact with everyone.
During April-May of 2024, we have seen incredible humanoid robot development, some are already on the market, and some will be in the next couple of years. Here some of the main ones I know:
Figure 01 from Figure AI: Multipurpose AI powered humanoid robot. “Figure 01 is AI-powered and self-reliant, ready to produce an abundance of affordable, more widely available goods and services to a degree which humanity has never seen.” LINK
Digit from Agility robotics: Digit is already in the field being used and tested by a handful of companies. Digit is one of the most advanced systems and one of the first humanoids to be ready for the market. Its main function is to move goods and boxes in warehouses (supply chain). On June 5th 2024, digit was officially active working on a warehouse of GXO Logistics Inc in Atlanta. This is not a test, but a real official use case of the robot in use. - LINK
Optimus from Tesla: Optimus is a general-purpose humanoid robot capable of performing tasks that are unsafe, repetitive, and/or boring. It is not clear if its main role will be in an industrial or domestic context. The company's bet is even bigger as they promised the robot would cost around $30,000 USD, and Elon said it is going to hit the market by the end of 2025.
In any case, Tesla's development of Optimus is incredible; they are moving much faster than many experts in the industry were expecting. Of course, they have a lot of experience in mass manufacturing, computer vision, and AI with their autonomous vehicles—knowledge that they are extrapolating to their robots. LINK TO X Profile
Atlas from Boston Dynamics: The electric version of Atlas was announced in April 2024, this version looks a bit different to the other humanoid robots in development, with its round “face” which is a very interesting design choice make it unique, well to me looks very similar to the small Star Wars mechanic droid. At the moment we do not know much from this version of atlas, but it has a lot of potential. We know what its predecessor version was able to do (Hydraulic atlas) which was one of the most agile robots out there, capable of running, jumping, and even to parkour. If the electronic version has some of this capacity, it will be able to perform many different tasks in any type of context, even outdoors. - LINK to Atlas Electric article
Apart from VLA, other technological breakthroughs are helping in bringing all these amazing robots to our world. Nvidia offers many innovative services, development platforms and specialized hardware that are powering many of the robots we are going to see revolutionizing our world.
Nvidia is well known for its hardware solutions, it offers a line of computer boards designed to bring AI at the edge. These solutions offer incredible computational performance with low power consumptions, making them perfect for robots, drones and other autonomous machines. Why is edge computing important? In a very simplified way, edge computing is when all the computational activities are performed “locally” inside the machine that is performing the task, without the need to be connected to the internet using the cloud computational power. This is especially important for autonomous machines that need to run very complex AI models and make decisions in a blink of an eye. It is important because it could be very dangerous if this autonomous system is performing tasks in uncontrolled environments with connection issues; they will need to keep performing without any connection.
But apart from the hardware, Nvidia is helping the industry with its developing platforms focused on robotics and autonomous systems Isaac and GROOT. Isaac was launched in 2018 with major updates in March of 2024 together with Groot’s launch.
NVIDIA Isaac: Is a platform that provides a suite of software, libraries, and frameworks, including tools for AI perception (computer vision processing), manipulation (action control), simulation, and software development kits (SDKs), which give developers the foundation to build robotics applications.
One of the most interesting tools that Isaac offers is the NVIDIA Isaac Sim, a realistic simulation environment used to train and test robots before being launched in the real world. In these virtual environments, robots can train their AI models, learning and improving quickly with no real physical risk. Isaac is also powered by generative AI, which allows robots to perform tasks that were previously difficult to program. NVIDIA also offers cloud solutions like Omniverse Cloud to help train AI and robotics platforms without the need for powerful hardware.
Nvidia GROOT: Is a general-purpose foundation model focused on developing a foundation model for humanoid robots, with the scope to create a core set of functionalities for humanoid robots, allowing them to interact with the world more effectively by enabling them to learn and adapt to new situations, perform more complex actions, and/or interact with humans more naturally. From the NVIDIA site: Robots powered by GR00T, which stands for Generalist Robot 00 Technology, will be designed to understand natural language and emulate movements by observing human actions — quickly learning coordination, dexterity and other skills in order to navigate, adapt and interact with the real world. In his GTC keynote, Huang demonstrated several such robots completing a variety of tasks. GR00T is still in development, and it should be available by the end of the year. It is very interesting that it is completely focused on humanoid robots, and an innovative learning approach of observing humans do tasks. Again, this confirms the importance of the humanoid trend, and how advanced AI tools are going to change everything.
These tools are empowering the development and design of intelligent robots, it seems that in the near future, you would not need to build a specialized team, expert on many technical areas in order to create a robotic solution, as many of the skills and knowledge needed will be available in a simpler automated way. Developing robots has never been this easy, maybe I am simplifying the complexity too much, but it seems that these tools give the opportunity for smaller non-technical teams to develop interesting autonomous systems. It could be the start of a small “democratization” of robotics.
Of course, I am not saying that technical roles are not going to be necessary anymore. On the contrary, they will be more important than ever, but with tools that make the process more easy and transparent, other professions could actively participate in the development of these types of complex products. Let's remember that in the past almost all types of robot development has been for research purposes mainly in academia, where the focus was to solve very technical problems. Now that many of the “baseline” problems have been solved (Like making the robot walk, recognize and grab objects, move with a load, etc.) companies will start focusing on bringing these solutions to the market, and other problems will need to be solved, problems that other types of professionals will need to solve. Of course as a designer myself, I see the importance of designers to be involved from the beginning of the development mainly for the ability to understand and empathize with the end user.
For example, the importance of applying design thinking and user centered design principles in the design process of HRI, not only on how the robot should behave, but how it should look and move to communicate the correct message to the users to make them feel safe and trust the robot. In the end, robots are just another product, but here understanding the real behavior, needs and desires of users will be much more critical and important than before. We are not talking about a bad interaction with an app, but it might be a life-dead situation if these robots are not able to communicate perfectly their intention to the users.
We will be experiencing new types of interactions that we have never seen before, we will for sure learn in the process, and we will need to find innovative tools and methodologies that will help us interact with these new technologies in the best way possible.
Social robotics
Shifting from technical aspects to the social and emotional facets of human-robot interaction, “Social Robotics”, focuses on creating robots that engage in social interactions with humans such as conversation, companionship and emotional support. The goal is to create robots that maintain meaningful relationships with humans.
This doesn't mean that social robotics cover only robots that their main scope is to create meaningful relationships with us, it can also be applied to other robots that have different scope/function. For example we could have a bartender robot (Like ADAM from Richtech Robotics) whose main function is to serve you a drink, but at the same time will have an interaction with you, that whole experience will need to be designed, curated and controlled.
Of course the focus is bigger on certain contexts where the interaction between human and robot will be more social and interpersonal, like robots that support the elderly or people with disabilities, or educational robots for children, or entertainment robots
The concept of “social robotics” was first used by Professor Cynthia Breazeal back in the 1990s when she was a graduate student at MIT, and she was working on robots capable of interacting with humans in socially intelligent ways. Professor Breazeal is a prominent roboticist who worked on social robots like Kismet (back in 1990) and Jibo (In the market in 2017). She has published several papers and the book: “Designing Sociable Robots" in 2002.
In her book, Breazeal explores creating robots capable of understanding and responding to human social cues. She discusses the challenges involved in designing such robots, including the need for advanced artificial intelligence, natural language processing, and expressive physical embodiments.”
Breazeal envisioned a future where robots will be like “synthetic creatures” capable of learning and growing from their experience, understanding human emotions and intentions, and collaborating with humans.
As was point out before, we are getting to this “future” professor Breazeal envisioned more than 20 years ago, with the advances on AI we are near to have this entities that seems to be able to “understand” our intentions and learn from us and from past experience, and they are getting pretty close to let us think they are “conscious beings”
In the book, there were 3 key areas of exploration:
Social Intelligence: Here it was discussed how the robot can communicate with humans, verbal and no-verbal communication. How the robot will be able to recognize human emotions, like facial expressions, tone of voice and body language. And how the robot will be able to learn and adapt from their social interaction with humans, changing its behavior depending on the moment and person they are interacting with.
Human-Robot Collaboration: Here the focus is a bit different to what we discuss before in our section of “Collaborative robotics”, in this case Breazeal talk about the importance of having a shared goal for the human and the robot, as well of the importance of the trust and rapport between both entities.
Social Integration: Breazeal explores how these robots will have a key role in our future society, she focuses on the educational and eldercare sector.
Breazeal emphasizes the importance of an interdisciplinary approach to achieve these goals. Fields like psychology, neuroscience, and robotics engineering all come into play to design robots that can truly function within a social sphere. There is no direct mention of the role of designers back there, but their importance in this field is implicit, as for a social robot to be successful, concepts like User Centered Design are crucial for these robots to be user friendly, aesthetically pleasant and adapted for the specific needs of the target audience.
Lately, Breazeal has been working on the evolution of the social robotics concept with the integration of new innovations in the field of AI. She talks about how these intelligent machines will help us grow, thrive and flourish as society. She believes these “AI companions” will soon arrive and be part of our day to day.
Breazeal explores the robot perception, Interaction, Learning and Expression with a Socio-Emotive lens, focusing on how these elements of the robot interaction will be different when the machines will be in direct contact and communication with humans.
In other developments, Disney and their amazing team of Imagineering, are working on several new social robots for the parks. Robots at Disney are nothing new, actually one of the main experiences for many years was the animatronic version of Adam Lincoln in their first park in California. Now they are working on new social robots from the star wars franchise that will be walking in the park interacting and socializing freely with the park goers. The mission of the development team is that the users will not see these small cute robots like robots, but like the real characters from star wars. This for me is the next step of social robotics. We are not talking about the creation of a robot that can interact with hundreds of people in a safe, fun way, but robots that are creating an experience so good and immersive that it will make you think you are in a galaxy far far away.
Conclusions
The design behind our relationship with robots is more important now than ever, and it should not be left just to the technical / engineering team. As we create machines capable of learning, creating, and making decisions autonomously, it’s crucial to design clear, intuitive interfaces for human-robot interaction. Our current relationship with robots, shaped by advancements in Human-Robot Interaction (HRI), Social Robotics, and Collaborative Robotics, is evolving rapidly due to breakthroughs in AI and robotics.
Disciplines like HRI and Social Robotics highlight the need for collaboration among designers, engineers, psychologists, and other professionals especially if the scope is to create robots that function effectively and communicate transparently with humans. The success of products like OpenAI’s ChatGPT shows the power of intuitive design. This extends to robotics, where integrating AI models like Vision-Language-Action (VLA) is transforming how robots perceive and interact with their environment.
The rise of humanoid robots from companies like Figure, Agility Robotics, Tesla, and Boston Dynamics marks a significant shift towards robots that can navigate and perform tasks in human-centric spaces. Platforms like NVIDIA’s Isaac and GROOT are democratizing robotics development, enabling more professionals to contribute to this field.
As we approach widespread robotic integration into daily life, focusing on user-centered design and transparent communication is crucial to ensure safety, efficiency, and trust. The future of robotics lies not just in technological advancements but in thoughtful design that facilitates seamless, meaningful interactions between humans and machines. Through continuous innovation and collaboration, we can shape a future where robots enhance our lives and integrate harmoniously into our society.
This article was written with the invaluable support of Daniela and Marihum for editing, fact-checking, and general advice.
Thank you for reading, and we look forward to seeing you in the next chapter.
Ciro Davila.
P.S. If you're eager to continue the conversation, consider subscribing to the Coexist newsletter. Don't hesitate to reach out if you're interested in engaging in discussions on these topics.