In Conversation With Maja Pantic

I

Research & Recognition

Prof. Pantic’s research has spanned several areas of AI, including computer vision, speech recognition, and affective computing. Maja has published numerous papers in prestigious academic journals and has received several awards for her contributions to the field. As of writing, Maja has an h-index of 98 and has more than 46,000 citations to all her publications which is truly exceptional.

Maja has received several awards for her significant contributions to AI and her work on facial expression recognition including the British Computer Society’s Roger Needham Award. She is named Fellow of the IEEE, and Fellow of the International Association for Pattern Recognition (IAPR). Maja was also elected the Fellow of the Royal Academy of Engineering.

In addition to her academic work, Maja is a recognized leader in the tech industry and an advocate for women in technology. In her TEDx talk at CERN, Maja continues to share her thoughts on how our everyday facial expressions can be put to good use. Maja also spoke at the World Economic Forum sharing her thoughts on how we can apply machines that can interpret human emotions “in the wild” and put them to use in different industries from medicine to marketing.

Overall, Maja’s contributions to the field of AI have been significant, and she continues to be a leading voice in the development of intelligent systems that can understand and interact with humans in a more natural way and not only depend on data sets.

Today, we are extremely happy to have Maja with us! Join us to discover her incredible life journey, the challenges she faced along the way, and the choices that led to where she is and what she has become today.

I mean, very honestly, the only thing that was on my mind as a child was wanting to be independent. It’s funny because I didn’t want to live with my parents even though they were very nice. I just wanted to live alone and be in charge of my own decisions and actions. That was very clear to me.

I have a vague memory from when I was three years old, my mom told me about it later. We were at the beach, and I took a small towel, put it on my shoulder, and told my mom that I was going out into the world. She didn’t take it seriously, but I started walking away until she couldn’t see me anymore.

But in terms of curiosity about what I was going to do, everything was interesting to me. I loved to read, especially fiction from different cultures like Japanese, German, and Russian literature. I read anything and everything that came my way.

Reading was something that I was good at in school, and it came relatively easy to me. Unfortunately, I never did very well in physics despite my best efforts. I even prepared for the entrance test in electrical engineering but eventually realized that I wasn’t good enough in physics. Instead, I decided to study pure maths. So that’s how it went.

Great! Looking back at your life and reflecting, what would you say were the turning points in your life?

The first definitive turning point for me was when I decided to attend the Mathematical Gymnasium. That meant I would be studying electrical engineering, mathematics, or related fields. This was a difficult decision because I grew up in Belgrade, Serbia, and until you are 15 years old, you have one math subject in school, along with physics, chemistry, and other subjects. However, when you attend the Mathematical Gymnasium at 16, you have seven different math subjects to study, including geometry, probability, logic, mathematical analysis, and more. In addition to that, you also study Physics, Serbian language, English, and Astronomy, but no chemistry or history. I think this was my first turning point.

I was never able to finish my university studies in Belgrade because we experienced war and political turmoil in Yugoslavia, which caused the university to close several times. Meanwhile, my sister moved to the Netherlands, and I decided to follow her and try to continue my studies there. However, when I arrived, I was informed that none of my previous courses would be accepted, even though I had been studying Pure Math for four years. This was in 1992, the year the internet was released.

And that was, in my opinion, the biggest turning point for humanity. The Internet is, in my opinion, the greatest invention not only of the 20th century but probably of the last 100 years. Sure, we wouldn’t have the Internet without electricity and all these other things, but the Internet opened up the world. There were no borders anymore. You could know what was going on everywhere in the world based on the Internet. At that time, I realized that what I needed to do was pursue computer science. I went to the Netherlands and said, “Okay, if you don’t accept anything I did in math, I will simply go and study computer science.” So I started that, and of course, because I was more mature and had already learned most of the things from math, it was very easy for me. I finished a five-year program in just three years.

Leaving everything behind at such a young age especially to pursue a new career path, sounds challenging. But in your case, was it just a logical decision because you believed that computer science was the way forward?

Yes. Just think about it, we now have something that can connect people across the world, regardless of distance or time difference. And I knew we were just in the initial stages of this technology in 1992. When I was in high school and university in Belgrade, I was already programming and had exams in different programming languages. By the time I came to the Netherlands, I already knew five or six different programming languages, so it was easy for me to choose computers and the world wide web aka Web1.

Wow, you really had a good vision and a sense of where the future was headed. Here we are heading to Web3!

Absolutely. The potential for communication and information-sharing was immense, and I was excited to be a part of it. So, it was clear in my mind that I wanted to pursue the field of computer science.

The next decision or turning point in my life was about whether to work in a nuclear power plant in the Netherlands managing their processes with machine learning or become a Ph.D. student. The proposed topic was computer analysis of human faces and behavior, which was a completely new subject at the time. I was the third person in the world to touch on this topic (after the teams at the UCSD and the CMU) and the first in Europe. It was extremely interesting to me because I have always had an aptitude towards understanding human behavior and helping others to understand each other better. Faces are like windows to what we think and feel and can reveal our internal state of mind. For me, studying faces was a fascinating area of computer science that allowed me to combine the human and computer worlds.

After completing my Ph.D., I became an Assistant and then an Associate professor at the Delft University of Technology. My husband later received a professorial position at York University in the UK, which is why I moved to the UK.

Being the third person in the world to pursue something is incredible. But being a pioneer also comes with its fair share of risks. For example, people often fear change, don’t they? Did society respond similarly to your work to study facial expressions? Can you provide us with some insights into this?

Yes, people did ask those questions. I found this topic extremely interesting because I was fascinated by robots. From a robotic perspective, the easiest way to communicate “no” or “don’t do that” is through facial expressions. First, the facial expression comes, then the words. At that time, automatic speech recognition was not as well-developed. Gestures can also be used, but facial expressions are often the most effective. For example, a surprised or fearful expression can alert a robot that it came too close. This was one of the things we thought would be interesting. Although this was realized later, we also soon discovered that we could use automatic facial expression analysis in medical studies to understand things like depression and well-being.

If these systems could be built to operate in a very private environment, it would be particularly beneficial. Although we couldn’t achieve this at that time, it’s now possible due to the availability of apps that can run on your device. The app can keep everything within it on your device and not share it with anything else outside of it. You can now have such an app to monitor your daily mood and suggest behavioral changes (like walking on the grass, calling friends, spending time with a pet).

It can be like an AI friend or an AI companion.

Later on, we discovered that a lot of diagnostics could be done based on expressions, such as identifying symptoms of conditions like Parkinson’s, schizophrenia, and even suicidal depression. We also found that dementia has specific facial signals, and even internal issues like liver problems could be diagnosed based on the coloring of the face and sclera. These are just diagnostic tools, like a CT scan or an MRI scan, but based on an RGB camera, not intrusions on privacy as some may think. While it’s not yet fully developed, this is something we should continue working on.

But the idea of communication with AI for the purpose of improving people’s well-being has been around for some time.

Indeed, there are so many potential good use cases for the technology, but it’s essential to use it responsibly. What according to you is ‘the Internet’ of the 2020s? And what are the challenges we can face?

The Internet revolution in the 90s brought significant changes and opportunities to the world. Now, we have new opportunities arising from AI, the metaverse, and other advancements. We have ChatGPT that arguably fully passed the Turing test and I think that this is huge. But this can either be a downfall or an uprise of the Internet. The reason I believe it can be a downfall of the Internet is because of the risk of pollution of the data we use to train these algorithms, as anyone can access and teach them. For example, there are people who believe the Earth is flat and that 5G is carrying coronavirus, and there are all sorts of religious fanatics who can feed misinformation to algorithms like ChatGPT. This can lead to misinformation and negatively impact the Internet’s integrity.

We must ensure the integrity of the Internet to prevent the spread of misinformation and pollution. Therefore, we need to focus on researching ways to maintain the Internet’s integrity. It is extremely important in my opinion, in fact, the most important immediate research topic that needs to be addressed. This might require regulation of the Internet to protect the digital world.

However, I also think it is extremely important for humans to have the right mindset. We should stop thinking about ourselves as belonging to nation A or nation B. We are all Earthlings and live on the same planet. Together, we have to protect our planet and our digital world. Regulation may be necessary to maintain the responsible use of the internet and protect its integrity, but these regulations should be decided on by all (Earthlings), not by just a few nations.

You mentioned a good point here. If we don’t take ethical responsibility for the use of the internet, it can be too late to clean the pollution at one point. How would technology help governments to improve their policies around this or how would technology help change the way the government works?

I really think that if we have to start somewhere, it should be with humans. Humans should simply realize that we are at a very dangerous point in our civilization; we are all dependent on the internet. Most of the recent knowledge exists in digital form only. A lot of us have information at our fingertips; we can press a button and get the information we need. But if that is polluted and we cannot rely on the Internet, we shall have a giant step backward as a civilization. The integrity of the digital world is therefore extremely important.

On the other hand, governments should realize that it’s not about just one government; it’s about everybody in the world. We need to transcend the notion of “government” and form a planetary alliance of the human species. The whole world is in trouble; we polluted the Earth and oceans with plastics, we polluted the space with debris, and now we have the means to pollute our digital world too. We will simply not be able to survive; it’s as simple as that. So, it’s no longer just one government’s problem; it’s everyone’s problem.

But I do believe that technology can help clean up the Earth and space pollution for as long as we keep the integrity of the digital information. Generative AI can give us suggestions on how to bring new energy and clean up pollution. For example, taking away carbon dioxide from the air and transforming it as a new energy source. But for Generative AI to suggest plausible solutions for such long-standing problems, we need to ensure the integrity of the internet and the digital information from which it learns. And then we all need to be united in wishing a better tomorrow for all our children and giving that the highest priority (rather than giving the profit a higher priority), not just in the UK or Sweden, but everywhere.

Absolutely, we are in fact more connected than we think. To put things into perspective, ocean debris from Japan can reach the coast of the United States within a year due to currents. It’s not just about the West or the East; it’s high time we think about the entire planet and take responsibility by acting ethically.

Exactly, this is something that I hope either the World Economic Forum or a body like that needs to take the lead on.

The power struggle is the worst sickness of humans, and we need to transcend it. Otherwise, we won’t be able to survive.

What is your technique behind dealing with challenges and what have been the biggest challenges that you have faced so far?

I don’t know what my biggest challenge was. I do not consider anything in my life to be clearly difficult. One can say that leaving my home country and moving to another country was a challenge. But I was young then and that was more fun than a challenge. I lived in Serbia for 20 years, and then in the Netherlands for 15 years, and then in the UK for another 17 years. Changing countries is not easy because you have to learn new processes, languages, and social norms all over again. However, doing it multiple times has made it much easier for me, and nowadays, I don’t consider it a problem at all.

If there is something I would mention as a challenge, it’s being a female in a male-populated tech industry. I was the only female MSc graduate in 1996 in Computer Science at the Delft University of Technology, the Netherlands. I was also one of two female professors (out of 300 professors) in 2001 in Electrical Engineering, Mathematics, and Computer Science at Delft University of Technology. Then I was one of only two females on the Board of Directors for Samsung Electronics (out of 160+) and one of very few female tech leads in Meta. In Meta, as well as in other tech companies and tech-oriented academia, there are some female product and project managers, and HR is mostly female. However, tech is very much male-dominated. Both Korean and American tech companies (and tech-oriented academia) need to change the way they choose their leadership; they are building technologies for the entire population while having the directorship chosen from only 50% of that population. That means that design choices will be very much biased towards 50% of the population while the tech is supposed to be built and used by the entire population.

How can we overcome this gender parity? As a woman in tech myself, when I speak to my male colleagues, I sense that most of them don’t see any problem with the current situation and that it is exaggerated. On the other hand, I see successful women who have worked so hard yet don’t receive the recognition they deserve. Women are still a minority in STEM. This is a fact. We spoke about working together as one planet, how can we work together as men and women? What should the solution be instead of operating as two groups in a vacuum?

One of the issues is nepotism, which is a human characteristic. Boys tend to be friends with boys, and girls tend to be friends with girls. However, girls are often afraid of nepotism. They will often not hire a female friend, but instead, consider male and female candidates equally in order to be fair. It’s not fair to say that girls should practice nepotism, too. Instead, we should make it clear to all that nepotism is a human bias and a subconscious choice that we need to avoid. I had managers who openly told me that I need to build alliances (with male directors) to get promoted. Overcoming this tendency and trying to employ, judge, and promote people based on their merits rather than nepotism should become a norm.

One way of achieving this is to build in a constant reminder. Whenever you have an interview, an annual evaluation, or a promotion round, take half an hour beforehand to remind yourself that you should not hire/ promote your friends but people who have merit and results. If you do not do this, you will end up with a skewed and biased group of employees, probably all having similar skills. No or little progress can accrue from such a group.

In summary, we need to be conscious of our biases and work towards creating diverse and inclusive workplaces where everyone has an equal opportunity to succeed.

I stand by the belief that women are not recognized. Look at the path that we go through. It’s always women who wait longer for promotions than men and who are paid less for the same job and function. Always. I think this is a pity, and I believe it’s because of the unconscious bias that we have.

I couldn’t agree more, I think it’s a very interesting topic to address and solve. Moving on to your work life – can you share a few insights about your day-to-day and what it’s like working at Meta?

Meta is great. I work on extremely interesting projects, such as building Avatars, which is very interesting to me. I also work on animating these avatars, which is another part that is super nice and interesting. Working at Meta is especially great because we have some of the best scientists and engineers who are really knowledgeable in what they do.

However, I am slightly disappointed with middle management, the people who sit between the highest management and the tech leaders. I consider myself a kind of technical director, and the people who are just above me, going to the next 2 or 3 levels, are very often just managers who have very little knowledge about technology.

They like to organize stuff, and while some are good at it, some managers lack the technical grasp and have no understanding of what we are talking about. So they often lack in vision, strategy, and blue-sky ideas. I think that’s a pity, and I think Meta has started to understand that. If Mark Zuckerberg can solve this a little bit, it would be great because it would make a difference for the company – we could move faster, in a better direction and the life of scientists and engineers will become easier. In principle, a lot of these middle managers let people under them do it all – come up with ideas, plan them, and execute them – and they take the credit. This should stop.

It is indeed troublesome to work with such leaders who barely understand the project, especially from a technical perspective. It is even more painful when they set unrealistic goals and targets to deliver. So according to you, what makes a good leader? What are the three key factors for better leadership skills?

A leader is not somebody who sits behind their team and simply delegates tasks. They take part in the responsibility and lead by example. ‘Walking the talk’ is extremely important. If you say something needs to be done, you should also be a part of making it happen.

In principle, everybody has a different skill set, and I believe good leaders will recognize that and give tasks to those who can do them well. I had male leaders of the companies who preached about diversity but then told me that I need to develop the exact skills that they have in order to be promoted – as if the company needs carbon copies of a few managers. This is a sad example of leadership. I value my team members for their diverse skill sets and assign tasks so that people can grow. I always advise the team on almost all matters and take tasks that would be difficult or tedious for them to do.

Sharing love is really important. It’s not just about directives. If you truly care for your team, you will value them for who they are and what they bring to the team. You cannot expect only all-rounders, especially on a high level. If you have a person who is extremely good and passionate about coding, you cannot expect that person to go and win negotiations. No two people are the same, so leaders must recognize people uniquely for who they are.

In summary, these are the essential things for leaders:

What are the potential use cases of the Metaverse? In which direction are we heading? Tell us more about it!

In principle, the Metaverse is a combination of real-world, augmented reality, and virtual reality. What I absolutely see as a future is that phones, as we know them now, will not be our future. They are cumbersome and heavy, and you need to carry them around as separate devices. It would be much easier if everything could be integrated into a pair of glasses. You could have them as a phone, a display of information, and an information provider on the go. For example, you could ask directions to a specific place. Instead of going online and navigating to your destination with the help of Google Maps, you could receive the answer through the glasses with arrows indicating where you should walk/ drive. I believe this is much more useful than the current reality, and I think this opinion is shared by a lot of people.

On the other side, virtual reality is a space where you can relax and connect with others. Instead of just talking to your family or friends on the phone or via video call, you could be transported to a virtual world and some beautiful surrounding where you don’t have to be physically present to have a conversation or hang out. For example, you could be on a beach or in a peaceful environment for therapy or just to relax.

Gaming is also a big part of virtual reality, and there are limitless possibilities for creating gaming spaces. Of course, physical contact with friends could still happen in the real world, children can go to school and play physical games, while they could also join their friends in a virtual world while still being in their own physical space. How soon we will transcend to this is a bit unclear at this point.

But there are quite a lot of interesting potential use cases for the Metaverse. In my opinion, the most important is in the medical field. Doctors could see us without us leaving our living rooms. If we are unwell, we could receive immediate attention from a remote doctor. For physical issues that require intervention, we could potentially have tubes that could be used for remote operations. This future reality would be extremely useful for medical care, in my opinion.

When do you think this will happen and what are the biggest challenges?

I think five years is reasonable. I don’t think it will be much sooner, both software and hardware have unresolved issues. Things like having these glasses with screens that are see-through and yet can display information and are lightweight are not easy to build. Having them process huge amounts of information in real-time while not overheating is another challenge. You have to take into account that current chips are 100 by 100 atoms, so they cannot be much smaller and do much more processing. So, we either go to quantum computing or redefine how these atoms will be connected and how they perform. I believe quantum computing is definitely another certain future because it can accelerate and improve digital processing.

How do you have fun and what’s your definition of fun?

I have fun with the people I love, such as my son and my husband. We spend a lot of time together doing things we enjoy, like diving and being in the nature. I believe that achieving a balance with the nature and becoming a part of it while being with loved ones, is an amazing feeling. So that’s the whole philosophy for me, exploring the world together while enjoying independence is at the core of everything.

What is success to you, and what is your advice to our readers for being truly happy and successful in life?

In my opinion, the most important thing in life is independence. It’s crucial to be self-sufficient and not dependent on another person. Being happy with yourself and your independence brings a sense of pride. Knowing that you can survive completely on your own and take care of those who depend on you brings freedom and an extremely strong feeling of satisfaction. What makes me happy in principle is being completely self-sustainable and taking care of my family.