What if a machine could read your dreams for you, decode the hypnotic visions of a psychedelic trip, or even give shape to the thoughts of a patient locked in a coma or on their deathbed? Suddenly some of the most mysterious and ineffable realms of human consciousness would become surprisingly accessible to us.
In fact, that fantasy might be a lot closer to reality than many of us think. Physics students from the Stevens Institute of Technology in New Jersey recently trained machine-learning models to reconstruct visual images from recordings of a person’s brain waves. They were able to determine when a person was looking at an image of a pizza, a panda, or another category of object. Others have done the same with fMRI, which tracks blood flow changes that occur when neurons consume energy, but EEG is vastly cheaper and easier to use. One costs in the low hundreds or thousands of dollars; the other is in the range of millions to purchase and thousands to operate. One can be taken home and worn to bed; the other is a massive machine confined to a lab or hospital.
Because the areas of the brain that create our visual experiences of the world are also used to imagine and to dream, the Stevens Institute students, Isaac Van Benthuysen and Jack Caputo, hypothesize that their approach could eventually be used to reconstruct entire scenes and thoughts that march through our minds when we’re dreaming or in altered states of consciousness. On May 8, the pair presented their work at Innovation Expo, a showcase of student projects.
I spoke with Van Benthuysen and Caputo, who were just finishing finals at the Stevens Institute, about the inspiration for their project, what makes using EEG so appealing for this kind of thought decoding, how it might be useful for PTSD, and the surveillance risks.
Read more: “Encounters with the Posthuman”
I read that you were inspired to do this study from a dream. What was the dream?
Issac Van Benthuysen: It wasn’t a single specific dream that inspired me. It was more just an overall general interest in my dreams. I’ve been having vivid dreams for as long as I can remember and keeping a dream journal for eight years.
How did the interest in your dreams relate to the study that you did?
Van Benthuysen: What we did was try to reconstruct what a person was looking at while they’re awake, but the brain uses essentially the same machinery to represent imagery in dreams and in reality, when it’s awake. Our idea was: If you’re able to reconstruct something that someone’s looking at in real life, you could use that same system to reconstruct what they’re looking at in a dream.
How closely did your reconstructions of images get to the original?
Van Benthuysen: We were able to get pretty good categorical accuracy. But there’s still a gap between the representation and the original image. That was our goal in the beginning—to try to get as pixel accurate as possible. But we found that EEG data just doesn’t have that amount of fine-grained detailed information to be able to directly reconstruct images. That’s not to say it’s impossible to create more accurate images. But no one’s quite figured out how to do it yet. And we have to try some different methods if we’re going to get there.
What are some of those methods?
Jack Caputo: The reason that EEG is really nice is because it’s more portable. But other groups are doing a similar thing with fMRI data instead of EEG data. fMRI can detect exactly where in your brain things are happening, so it has much better spatial resolution, more of that data that’s useful for the reconstruction. So the reconstructions are a lot better. The downside is if you want to use this in the real world, you can’t have fMRI machines everywhere, so the EEG portability is very nice.
If we were somehow able to make the EEG data and fMRI data represent the same thing—we’d use EEG data for its portability, but then match it against an fMRI model that we’d train ahead of time. You might be able to match the good reconstructions from fMRI with the portability of EEG. That’s one avenue I know some research groups are looking at. Instead of changing or improving EEG or looking at developing a different brain data technique, they’re using existing ones and trying to use little tricks to improve the accuracy.
Why is portability important for this kind of technology?
Van Benthuysen: It just makes it so much more accessible. fMRI is a humongous multimillion dollar machine, whereas EEG is basically a fancy hat. We were using it in the library. There’s absolutely no way we can bring one of the fMRI machines into the library. It’s also a lot cheaper. If you want to be able to reconstruct dreams, doing it with EEG is a lot more practical. You can sleep in it. Or if this were being used to help locked in or paralyzed patients, wearing EEG all the time is much more feasible.
You were able to reconstruct visual features like color, texture, and shape from the patterns in neural activity. What other kinds of visual and non-visual information can be gleaned from these kinds of brain-wave patterns now? Or is that for the future?
Van Benthuysen: It depends on where you place the electrodes. We used electrodes on the back of the head, because that’s where all the visual processing stuff is. But temporal electrodes would be better for decoding speech.
You mentioned locked in patients and also decoding dreams. What other kinds of things would this technology be most useful for?
Caputo: I like to imagine that if this ever gets enough traction to be employed in the real world, we’ll find new uses for it that we wouldn’t even have imagined before. It’s just a new way of being able to see in the brain, so it could lead to other discoveries or a deeper understanding of exactly how the brain works—if not on a physical level, then on a more phenomenological level, where it always gets messy. I’m hopeful that it’ll grow.
One other possible use is treatment for PTSD. Today a popular therapy for the condition is dream therapy, or investigating nightmares, and this EEG decoder could also be useful for that. Adding something like this where a therapist would actually be able to see a person’s dreams or a person would be able to get more accurate dream reports that aren’t just limited to their memory could definitely help improve the therapy.
Read more: “The Implant That Can Control Your Brain”
The Stevens Institute summary also mentioned that it could be used to provide a visual account of psychedelic journeys, and even the moment of death.
Van Benthuysen: What’s interesting about this approach is that we’re very limited in how we can study subjective experiences scientifically today. We can’t make any objective measurements of internal experiences, but if we’re able to decode the visuals that a person is experiencing, that could open up new ways to study subjective experiences.
What would it take to get from where we are now to the capacity to reconstruct complete thoughts and thought sequences or dream sequences?
Van Benthuysen: Reconstructing thoughts, like word-based thoughts—I don’t feel as confident speaking on that, but perhaps more dense EEGs that have closer together electrodes so that we can get better spatial resolution. That’s part of the main problem: EEG has such low spatial resolution— that it’s hard to get the detail we need to make really accurate image reconstructions. But new types of brain imaging are in development that have better spatial resolution and are less cumbersome than a giant fMRI machine. The models that we make are limited by the data that we get, and the data is limited by the technology that we have.
What are you working on next?
Van Benthuysen: I’d like to continue making improvements on this. Our current model targeted the overall color of the image more than other properties. One neuroscientist suggested that we might have better luck trying to identify forms, like the edges of objects. I want to try changing the target that we’re going for to a grayscale image and emphasizing the form. Also, we’re using one of the original AI image generation models, stable diffusion, but we want to make our own image generator.
Are there any risks to the way this technology might be used?
Van Benthuysen: In a surveillance type of way? Yeah, that’s definitely a possibility. I’d hope it doesn’t come to that.
Caputo: We already have invasive brain-to-computer interfaces funded by really rich people with maybe not the best motivations, like Neuralink. EEG represents a non-invasive brain-to-computer interface, which has a lot of advantages. But the principle is the same: If you’re able to tap in and get somebody’s thoughts, if you ever reach a point where it works well, it could certainly be used for surveillance. It’s not completely out of the question. 
Enjoying Nautilus? Subscribe to our free newsletter.
Lead image: metamorworks / Adobe Stock
