Artificial intelligence systems can now create remarkably accurate reconstructions of what someone is looking at based on recordings of their brain activity. These reconstructed images are greatly improved when the AI learns which parts of the brain to pay attention to.
鈥淎s far as I know, these are the closest, most accurate reconstructions,鈥� says at Radboud University in the Netherlands.
G眉莽l眉鈥檚 team is one of several around the world using AI systems to work out what animals or people are seeing from brain recordings and scans. In one previous study, his team used a functional MRI (fMRI) scanner to record the brain activity of three people as they were shown a series of photographs.
Advertisement
In another study, the team used implanted electrode arrays to directly record the brain activity of a single macaque monkey as it looked at AI-generated images. This implant was done for other purposes by another team, says G眉莽l眉鈥檚 colleague , also at Radboud University. 鈥淭he macaque was not implanted so that we can do reconstruction of perception,鈥� she says. 鈥淭hat is not a good argument to do surgery on animals.鈥�
The team has now reanalysed the data from these previous studies using an improved AI system that can learn which parts of the brain it should pay most attention to.
鈥淏asically, the AI is learning when interpreting the brain signals where it should direct its attention,鈥� says G眉莽l眉. 鈥淥f course, that reflects in a way what that brain signal captures in the environment.鈥�
With the direct recordings of brain activity, some of the reconstructed images are now remarkably close to the images that the macaque saw, which were produced by the StyleGAN-XL image-generating AI. However, it is easier to accurately reconstruct AI-generated images than real ones, says Dado, as aspects of the process used to generate the images can be included in the AI learning to reconstruct those images.
With the fMRI scans, there was also a marked improvement when the attention-directing system was used, but the reconstructed images were less accurate than those involving the macaque. This is partly because real photographs were used, but reconstructing images from fMRI scans is also much harder, says Dado. 鈥淚t鈥檚 non-invasive, but very noisy.鈥�
The team鈥檚 ultimate aim is to create better brain implants for restoring vision by stimulating high-level parts of the vision system that represent objects rather than simply presenting patterns of light.
鈥淵ou can directly stimulate that part that corresponds to a dog, for example,鈥� says G眉莽l眉. 鈥淚n that way, we can create much richer visual experiences that are closer to those of sighted individuals.鈥�
bioRxiv
