Goals and Approaches for a BCI Model Stack
Our invitation to the brain decoding community
At e184, we’re building next-generation neurotechnology for the next stage in human development: our collaboration with increasingly advanced AI. That means building a non-invasive brain-computer interface with unprecedented resolution, a device that can be mass-produced to let billions keep up with a fast-paced future.
Such a device will need high-quality magnetometers or other advanced sensors, to resolve signals from neuronal currents as close to crystal-clear as possible. But high-quality signals are only one part of the puzzle. We will also need to be able to decode them.
Brain decoding has seen enormous advances in recent years…and faces unique challenges. Like many other scientific fields right now, it is an area where machine learning has made an outsized contribution, with powerful new models that map brain data to language, images, emotions, and movement. At the same time, there are obstructions that make quality models much more challenging to build than in fields that have seen dramatic public progress, like language processing and protein structure.
We have identified some of these challenges, and found candidates for promising solutions. We are just beginning our journey. We don’t claim to have all the answers yet. Our aim in this post is to establish a dialogue. If you work on brain decoding, we’d like to hear your thoughts. Are there key requirements we missed? Better methods than those we’ve identified?
We are establishing our hardware lab soon. We plan to launch an AI lab next, to start building the model stack we will need. We will need skilled researchers from day one, to plan and lead the project. If you’d like to get involved, or to collaborate in another capacity, we’d love to hear from you.
Key Challenges
Compared to other paradigm applications of machine learning, brain data is uniquely challenging to model. Brain data varies significantly in the same individual over time, and it can vary dramatically between individuals. The brain is in many respects expected to be an non-ergodic system, one which does not return to the same behavior over time, and as such with behavior that cannot be captured with a finite statistical sample. As one of the most important metrics for the success of a machine learning model is its generality outside of its training data, this variability is concerning. It isn’t a show-stopper: there clearly are general dynamics in cognition, not just across humans but even shared between species.
Our goal, then, is to find a model architecture and gather training data that lets us leverage universals and adapt to changing brains over time. We do not need to do this in perfect, scientific detail, but we need to do it well enough for our initial product goal of a mental smartphone: a device that lets a subject produce text and images with the power of their mind. We want that device to be a foundational technology, one that lets us build toward the full potential of a brain-AI interface for enhancing human cognition, and we know our data and architecture demands will grow and change along the way. But for now, we want a model stack that can cross that first hurdle: text, and images. The question then becomes what such a stack would need to accomplish.
If consumers are going to buy mental smartphones, they need to be comparable to, or better than, ordinary smartphones. That means comparable speed for text, 80 words-per-minute or better, and ideally to speech, 150-200 words-per-minute, as well as a low error rate, under 2%. For images, the question of error rate is inherently subjective: do people feel that the device better captures what they imagine than simply writing a text prompt for an image model? One could imagine measuring this through a ranking leaderboard, like those used for chatbots today.
A mental smartphone should also not be too difficult to set up. As anyone who has helped an elderly relative use a conventional smartphone can attest, the more time needs to be spent learning a new technology, the fewer people will be able to adopt it. In a brain-computer interface, the challenge is that not only will users need to learn the system: the system will need to learn the user. Each brain is different, and brains change over time. Calibration will be needed, and it is important to keep the burden on the user of that calibration as low as possible.
Finally, in a project of this scale, it is important to coordinate resources. That means keeping track of costs in data gathered and computational resources used, finding methods that make efficient use of both. It means building models that can be reused, transferred to different sensors, different signals, or different populations. And it means keeping track of feedback cycles, how long a research avenue will take until it can be assessed and how much interplay needs to occur between hardware and model development.
Promising Leads
With these challenges in mind, we can begin to ask: what kind of model “stack” would address them? What pieces will we need?
Foundation Models
One avenue that is driving a lot of excitement at the moment is foundation models for the brain. Much as large language models are trained on text completion but can serve as foundations for models that perform a variety of reasoning tasks, the expectation is that a large model trained via self-supervised learning on brain signals (using masked prediction or contrastive learning, for example) would be able to serve as a pre-trained foundation for models to decode a variety of types of brain data.
In EEG, LaBraM is an example of what such foundation models are capable of so far, with impressive performance on multiple types of tasks. Currently, no model of this scale exists for MEG, where state of the art performance on language so far has had to leverage substantially smaller datasets. These results do indicate that performance continues to improve as datasets get larger. Between those initial hints, and the general success in foundation models at generalizing across a wide range of applications, we see a foundation model as a strong avenue to achieving our performance goals in words-per-minute, error rate, and subjective rankings of images. Creating such a model will require a substantial investment: in recording new MEG data, in computational costs, and above all, in time. But we can expect that investment to pay off, creating a flexible foundation from which we can more easily accommodate new decoding tasks, calibrate to new subjects, and adapt to developing sensors.
Lightweight Personalization
It will be crucial to be able to calibrate to new users efficiently, without asking users to spend too much time in calibration or accumulating too many errors when varying between users. To address this challenge, we will need a lightweight way to personalize models without needing to change deeper features. Ideally, we would want to keep a high-quality universal encoder, for example one built on a foundation model as described in the previous section, and then make small modifications for each user, adding a small number of additional layers or a brief fine-tuning. A lightweight approach of this kind would also allow us to make more efficient use of data, as performance typically improves more by adding more data per subject than by adding more subjects, so keeping the need for training data for inter-subject calibrations low will allow for a more performant model from the same total recording-hours.
A variety of approaches to do this are actively being explored in the literature, and several look promising. Broadly speaking, one wants a form of transfer learning, in order to reuse data from similar subjects on a new subject. One interesting approach would be to learn a “subject embedding”, analogous to the word embeddings used in language models, to capture differences between subjects in the embedding space. There are a number of distinct methods for aligning data between subjects, several of which have yielded substantial performance improvements. Personalization approaches are already showing their worth for several of our goals of interest, with an approach similar to participant embeddings used for language and with tuning to subjects playing a crucial role for images. Overall, while it is not immediately obvious which method in this area will be most useful, it is clear that there is already an ample foundation to build from.
Continual Learning
Brain data changes not just between subjects, but in the same subject over time. A useful device will need a way to adapt to its user over a variety of timescales to remain effective. While one could do this by periodically asking the user to run new calibration sessions, this would represent a substantial burden on the user and would lower adoption of the technology. Ideally, we would instead be able to gradually adapt over the course of everyday use, shifting the model appropriately without a dedicated calibration session.
This kind of “continual learning” has been proposed in the literature. An impressive recent example is EDAPT, a framework that achieves update times of 200ms on current consumer hardware. This approach will likely require gathering additional training data for the initial model, in the form of data from the same subject over multiple days, but these resource requirements don’t appear excessively onerous.
Affective Feedback
A smartphone doesn’t have to know your opinion of it to function. Generating images from mental data, though, will be a different story. We do not imagine images in nearly as much detail as we perceive them, so the results of image decoding are on some level necessarily subjective. In order to achieve our performance goals for images, we will need to capture that subjectivity.
The most straightforward way to do this would be a manual rating system, but for a brain-computer interface, a more powerful option is possible: rating images directly from brain data on the user’s emotions. Emotion decoding is already a fairly robust subfield, and has even been applied to image classification already. Affective feedback may also help refine accuracy for language decoding, though this is likely to be a more minor effect.
Generative Models
Ultimately, brain data will need to be “fleshed out” on some level to produce specific images or words. An interesting possibility is to work generative models into the mix for selected tasks, cleaning up language and clarifying imagery. There has already been some success along these lines in the literature. Generative models have been used to improve the fluency and plausibility of decoded text. Their role in imagery is even more essential, and essentially every successful approach in that domain uses generative models. We expect a generative model layer to be a useful feature in any approach that can reach our performance goals for speech and for language.
Artifact Removal
Under real-world conditions, brain data is nearly always contaminated by artifacts. This is especially true for EEG, but it remains true for MEG, especially as new sensors make MEG viable in more varied conditions. In order to reliably extract meaning from brain data and meet the lower error rate we aim for, we will have to reliably filter out the effects of eye blinks, head muscles, movement, and magnetic noise from nearby devices.
There are several techniques in the literature to remove artifacts. A recent interesting example is the Artifact Removal Transformer, which trains on several types of artifacts in order to remove them from the data. A method like this looks to be an efficient way to remove particular kinds of expected data artifacts, yielding more accurate and robust systems.
What do you think?
We welcome feedback. Our method tends to be one of specialization: finding a few high-priority challenges and approaches, and focusing on understanding them better. Are there important challenges we aren’t considering? Powerful methods we’ve overlooked? If you have something to add, please comment here, or reach out to the Director of our Neurotechnology Program, Peter Zhegin, at p@e184.com.
We would like to thank Blake Richards for discussions which sharpened the ideas in this piece, including helping us reach a better understanding of the unique challenges of brain data and the space of promising approaches to working with it.