Jean Mary Zarate: 00:05
Hello, and welcome to Tales From the Synapse, a podcast brought to you by Nature Careers in partnership with Nature Neuroscience. I’m Jean Mary Zarate, a senior editor at the journal Nature Neuroscience, and in this series we speak to brain scientists all over the world about their life, their research, their collaborations, and the impact of their work.
In episode five, we meet a researcher devoted to understanding the complexities of vision and how to bring eyesight back to the blind.
Pieter Roelfsema: 00:40
So my name is Pieter Roelfsema. I’m the director of the Netherlands Institute for Neuroscience in Amsterdam. And I’m also a professor at the Amsterdam University Medical Hospital, and at the Free University of Amsterdam.
I studied the visual brain since many years, and I always wanted to know what happens if you really start to understand what you see.
And that’s far from trivial, because actually, what you see is just many items, many small image elements that fall on your retina.
But then you have to kind of group that together into a representation of objects and the several objects that may surround you. And that’s a very fascinating subject.
So I studied that for, for many years. And in the last couple of years, we found ourselves implanting an increasing number of electrodes (and electrodes is just another word for wires) into the brain of experimental animals.
And at some point, we realized we are at such a high number, hundreds, let’s ramp it up a little bit.
So we went to 1000, with the idea of stimulating those electrodes and then creating artificial vision.
Pieter Roelfsema: 01:56
So I think we have several goals. First, is really to understand how vision works. And one of the topics that I’m particularly interested in is attention.
So of all the things that surround you, you can pay attention, pick out one of those items, what happens in the brain when you direct your attention to something.
And another thing is what happens if something enters into your conscious awareness? So what are very exciting questions, so some things may get into your awareness and some, some don’t. So we are also seeing that.
And one of the more applied goals of the lab is to create a visual prosthesis, a visual brain prosthesis, so people who lost the function in their eyes, the idea will be just to skip those malfunctioning eyes.
And to plug in the visual information from a camera. It’s one of the central centres, so centres for vision in the brain.
So we approach that from several angles. So one is to do modelling. Second is to study vision in humans, just having them respond with a button press, understanding what they can and cannot see.
But we also look at the brain mechanisms in experimental animals. So we look in mice, and we also look in monkeys.
Pieter Roelfsema: 03:20
So I started to become interested in the brain after reading a book by Doug Hofstadter. It’s, I think it’s a famous book. It’s called Gödel, Escher, Bach, and was very famous when I was starting to study in the ‘80s.
It was a gift from my father. And when I read it, I knew what I wanted to do, I wanted to study consciousness. So then, I actually started to do some projects just in my hobby time, first on snails, then on rodents. And at some point, I realized I want to study this in an animal that is closer to humans. And I will really understand what happens in the brain when we become consciously aware of something.
So we, in my lab, studying cognition and the role of attention and how it’s modulating the activity in the visual brain, we started to implant increasing numbers of electrodes. So electrodes are just wires.
And so, at some point, we reached a standard preparation where we implant, say, 200 electrodes, and then we thought, you know, we could multiply this with a small factor. And then we know that from previous work, that if you stimulate one electrode electrically, you’re artificially activating these brain cells close to the tip of the electrode. And a person or an experimental animal (it can even be a person who has been blind for more than 10 years), they’ll see a dot of light. And that’s with only one electrode.
So if you have 100 or 1000 electrodes, you can create 1000 of these dots of lights, because phosphines. And since the region, the visual cortex, where we implant these electrodes, have a map of space, where you stimulate in a map the subject sees at the same location, the outside visual world, this dot of light.
So if you have 1000, you can basically work with them like a matrix board, they can know from the stadium or from the highway. So if you of course, if you flash up one bulb, the person is going to see, well, a dot of light. But you can create patterns. And that’s what we set out to do.
So we’re basically writing to this matrix board that is in the brain, and, and see whether the animals are able to recognize them. These, these things that we write as patterns. And we found indeed that this is the case.
So we were able to write, for instance, we trained, we did this in monkeys. We trained them to recognize letters.
And so they knew that if they saw letter A they would have to make an eye movement to the above. If they saw the letter B, to the left, and so on and so forth.
And at some point we trained them visually so these animals were not blind, they could see.
At some point, we took the visual stimulus away, and we just wrote directly letter A to the brain.
And we were very excited to find that they were indeed making the same response, as when we would have presented the same letter visually. We published that in 2020. So one-and-a-half years ago.
So in our team, we need a lot of different expertise. And some of those types of expertise are within our own team. So we are knowledgeable about how to put wires, electrodes in the brain.
But we also have many collaborations with experts around the world, people who really know about how to make these electrodes so that they don’t damage the brain tissue too much.
We work with people in artificial intelligence who help us to take our camera, to take camera images and translate them into brain stimulation patterns.
We also collaborate with neurosurgeons who can inform us how to really make this device and make it something that is going to be feasible for a neurosurgeon to really implant in the brain because that is definitely a very important goal for me, to bring this to a patient.
So the visual procedures will be composed of several components. The first is a camera. You can use the camera that you buy. So there are now several companies that make these glasses that contain a camera.
And these camera images are sent to a small computer. It can be the size of a phone. And this will take in the camera image and create from it a pattern to be a post on the matrix boards in the brain.
Then right now, we still have really a physical connection between a connector that is implanted on the skull of the subject. It can be a monkey. It can be a human. We would like to make this wireless, so it will be a wireless interconnects with a brain chip.
And then from the brain chip, there will be several wires running into the brain. So these are the ones we call electrodes. And so, based on the image that the camera captures, there is this brain stimulation pattern.
And that then gives a rudimentary form of vision. So you’re not going to see full colour, full depth as normal vision would give you. It’s going to be very rudimentary, like you’re walking around with this huge matrix board in front of you, right? So it’s, it’s definitely not going to be superb, but it’s probably also going to be much better than nothing.
Yes, eyesight works. It starts, of course, all in the retina, that’s at the back of the eye, which is a very, very sophisticated device.
So there are large groups of researchers that are studying the retina. And then from there, the information is transported to the brain through the optic nerve. And then it starts in the cortex in the first region, primary visual cortex.
And there are cells, brain cells, neurons, that are selected for fairly simple features of the outside world, say the location and the orientation of an edge of light, whether that’s a vertical edge or a horizontal edge.
And so they really do a very local processing. So you have many of these processors and in parallel. So one would be straight ahead, one would be just adjacent, one would be in the upper left corner for every location in the outside visual world, there is a set of neurons that just care about what’s going on there.
And then if you go to higher regions, then this information of these individual detectors is combined in more and more sophisticated ways. So basically, what happens there is that you go from pixels, to concepts. And there are now many people modelling this.
So there also has been, of course, an artificial intelligence revolution that helped us understand how to go from pixels to concepts. And what these people in artificial intelligence find and how they model this process actually, is quite a good approximation of what’s going on in the human brain.
So also, in the human brain you have all these stages that are involved in this translation from pixels into concepts.
Now, seeing what the concept in front of you is, whether it’s a bicycle or a chair, that’s only one of the functions of vision, it’s not the only one. You also can steer your motor behaviour. And there are other brain areas that are involved in that. So they actually localize the edges. So if you want to pick something up, you need to know where your fingers are going to touch the object that you want to pick up.
And you need to know where it is, you need to know how to position all your joints, all your joints. And so all these transformations, they are also in part informed by vision. So that’s another very important role for vision to play.
Pieter Roelfsema: 11:42
So one thing that is, I think, exciting in this domain is the possibility now to occasionally record also neurons from human patients, and some researchers in my lab are doing this.
So these are patients who have severe forms of epilepsy. And the neurologist doesn’t find the right cocktail to suppress these epileptic attacks. So then the neurosurgeon comes into the play, and in some cases it is obvious what is the problematic region of the brain, but there are some occasional situations where the neurosurgeon isn’t 100% sure.
And then these patients get the set of electrodes, a set of wires in their brain, for about two weeks. And we have ethical approval then to attach to these clinical electrodes, very tiny wires.
And through those, we can record single neurons. So that was a method that was developed by Itzhak Fried several years ago. That gives you the unique opportunity to also record from brain cells that are tuned to specific individuals.
So if you record from these single neurons, you can also do amazing things. So other people, but also in our lab, sometimes you can, for instance, make associations between stimuli.
So suppose that, while you associate a famous person, say Jennifer Aniston, with another famous person, say, Barack Obama, then we demonstrated that if people recall these associations…so you give them a picture of Jennifer Aniston, and you ask them to recall what was associated with them, then you have some neurons that only respond to Barack Obama. And then they will become active the moment you talk about Jennifer Aniston, and ask them to recall this association.
So these things I’m also very excited about. Because the neurons that code those concepts are often also immediately the concepts that are actually in your consciousness, these are the things you’re thinking about.
So that gives you a very close link to what is really on the subject’s mind. And what you can see in the activity of neurons, which I find fascinating.
Yeah, so we are not recreating the eye. So we’re just skipping it. So I think that’s also why the vision that we’re going to produce is, it’s just much less quality than the normal vision.
Because we are implanting electrodes in the individual brain, and if we stimulate those, we activate a set of neurons that would normally never be activated in that constellation, That gives you just a dot of lights.
And it doesn’t give us the possibility to create different colours, for example, because neurons that are selected for different colours are intermingled, and you cannot just selectively only activate the green cells, or the blue cells, or red cells. So that’s why it’s somewhat rudimentary.
But the challenges if you kind of realize that you’re never going to be as good as normal vision, then are to get a good coverage of the visual fields because of the nature of the map of the outside world in the brain.
You have to realize that the primary visual cortex, which is the first region where the information comes from, the visual information that is processed in the cortex is huge. It has a surface area of 25 square centimetres on the left, and another 25 square centimetres on the right.
And to get wires everywhere in that region, which is also quite folded, is going to be challenging. So that’s one of the big challenges that we’re thinking about, how to make sure that we cover the map with electrodes.
If you only cover a small part of the map with electrodes, then the subject is only going to be able to see in a small region of the visual field, then they will be blind at all other locations. That is quite undesirable.
Another big challenge is to make an interface with electronics in the brain that has a sufficient longevity.
So we are now at the moment using so-called Juta electrode arrays. So these are arrays of stiff silicone shanks, we call them. So like, like a bed of nails, is basically what it looks like.
And we know that they work, typically for a year, maybe a little bit longer. But you know, you don’t want to implant a patient with a prosthetic device to find out that after one or two years these electrodes are encapsulated by glial cells. So basically the fibrosis, fibrous tissue that encapsulates the electrodes, and you lose the contact with the nerve cells. So in that case, you cannot efficiently stimulate anymore.
So that’s another challenge. We have to find electronic materials that have sufficient longevity. So if you implant them today, they will still be working, say, in five years, or 10 years, or even 15 years. I think these are two major challenges.
Pieter Roelfsema: 16:58
So I get the occasional request. And I have to explain to those people who contact me, this is not a clinically approved device. So it’s research. And our ambition will be to go to humans in the next say, two years, or maybe a little bit later.
But in that case, it’s still going to be research. So don’t expect from us in the coming five years a treatment. It’s just research. And of course, the research is very important because it’s going to help us make the next step, and go towards a device that is clinically approved.
Before we are there there are all kinds of regulations, which are there for a good reason. And we have to show that we comply with all these regulations.
With the technologies we’re using now, it’s always going to be rudimentary. But I would be very excited if you’re able to create a prosthesis that has, say 1000, or even 10,000, or even 50,000 pixels, should realize that your eye has 1 million pixels.
So if you count the number of fibres in the optic nerves, it’s about 1 million. So 50,000 is what we might aim for at some point. Sounds ambitious, but it’s only 5% of the normal, of the normal eye. And it’s, that’s going to be challenging, but if I would look back on my career, and we would have been able to create a device that has 50,000 pixels, and several people are using it and it’s catching up, I will be tremendously satisfied about it.
Jean Mary Zarate: 18:51
Now that’s it for this episode of Tales From the Synapse. I’m Jean Mary Zarate, a senior editor at Nature Neuroscience. The producer was Don Byrne. Thanks again to Professor Pieter Roelfsema. And thank you for listening.
