Amazing work from Clay @kosonocky.bsky.social, Alex Abel and LEAH labs, and all the collaborators and participants in the Bits2Binders AI CAR-T therapy protein design competition. The results are in!
bsky.app/profile/koso...
Results from an impressive world-wide binder design competition.
If you want, you can check out the data yourself! We made it as accessible as possible :)
github.com/kosonocky/bi...
Link to the preprint below!
www.biorxiv.org/content/10.6...
And finally, huge thank you to all of participating teams and competitors whose designs were the foundation of this effort! ❤️
And huge shoutout to my amazing co-authors Alex Abel, Aaron Feller, Amanda Cifuentes Rieffer, Phillip Woolley, Jakub Lala (@jakublala.bsky.social), Daryl Barth, Ty Gardner, Prof Steve Ekker, Prof Andy Ellington, Wes Wierson, and Prof Edward Marcotte (@edwardmarcotte.bsky.social)
This competition was made possibly by many fantastic collaborators and industry partners. Huge thanks to LEAH Labs, @adaptyv.bio, @twistbioscience.com, TACC, @modal-labs.bsky.social, Lonza, ScaleReady, VWR, KUNGFU.AI, Maker Clinic, Synthia, Nucleate AI in BIotech, and the BioML Society
We hope this is useful for the protein design community! We also provide extensive detail on how the competition was organized, a list of all 400 metrics, and all competitor methods in detail
In summary, we find that most of the failures seem to have been caused by impaired translation and protein expression. We believe that optimizing sequence-level properties for expression is just as important as the structure-centric task of binding
In contrast, filtering out <0.50 Boltz-1 ipTM only marginally increased the recovery by 0.7% and proliferation by 0.3%. This removes two non-functional designs from the top 10, but also removes two broadly functional designs with binding affinity
If we removed seqs with:
≥60% GC and <1.9 DNA entropy
≥45 AA repeats (DNA)
≥8 EE repeats
≥30% K+E alpha helix
We would remove 4,600 designs, increase recovery from 57% to 81%, and CD20-specific proliferation from 5.9% to 7.6% while removing 2/3 non-functional top 10 designs
We then find that inclusion of cysteine often caused the binders to undergo a *decrease* in proliferation. We think this is potentially due to misfolding and mispairing
That said, there was a sweet spot in K+E alpha helices for proliferation, which may simply be indicative of MPNN use, and the fact that those models 1) are generally successful at design, and 2) frequently choose A, K, and E
Literature also finds that adenosine repeats in DNA can cause issues with translation, as these sequences begin to look like poly-(A) tails. These sequences are rich in K+E, and lysine is encoded with AAA and AAG, and glutamate sometimes with GAA, suggesting a possible mechanism
But why are K+E alpha helices bad for recovery, CAR expression, and T cell viability? Previous literature has found that glutamate repeats of nascent chains can cause spontaneous ribosomal abortion. We find that the failed sequences are enriched in EE repeats.
Then, we find that these alpha helices are extremely enriched in lysine and glutamate, with over 35% of the residues in the design being contained in an K+E alpha helix. This feature alone obtains 0.89 ROC-AUC when used to train a LR to predict recovery across all 12000 sequences
First, we find that the failed MPNN designs were almost entirely alpha helices in their Boltz-1 predictions
Examining the effect of ProteinMPNN and SolubleMPNN (MPNN), we see that MPNN was able to both recovered and non-recovered designs, suggesting that failure was due to a particular behavior of the model in certain circumstances.
We then computed over 400 metrics on the sequences and their predicted structures. First, we find that many of the designs that couldn't be synthesized as DNA were repetitive with high GC content, in accordance with common DNA synthesis filters.
It's worth noting that each team's methods were multi-stage with many components, and that this high-level categorization may fail to capture causality
ESM2 has the inverse of this effect on recovery, molecular dynamics nearly doubled the success rate, FastRelax/AmberRelax had no effect, and "iterative diffusion" had no effect, despite its use by the top teams
ProteinMPNN or SolubleMPNN slightly increased the hit rate, but that almost 55% of these sequences failed to yield a functional CAR-T cell (called "recovery" for short) whereas almost all designs made without these tools (see below!)
We then analyzed the structure prediction models and found less meaningful correlation here. ESMFold is ranked first largely due to its use in the BAGEL pipeline, but the other teams that used it had lower success.
Our analysis focused on the outcomes of the proliferation screen. Of the methodological choices, we find that BAGEL was the top-performing generative method, followed by Chroma, RFdiffusion, and BindCraft
At the outset of the competition, we collected summaries all of the competitor's methods and wanted to figure out which methods and sequence features led to success (and failure). You can read these in Supplementary Information E
In summary, we have confirmed that at least three teams were able to create broadly functioning CD20-specific immunotherapies using the AI-driven tools of their choice. We gave our awards to these teams and others based on the functional assays released last September.
We partnered with @adaptyv.bio to measure the binding affinity of the isolated 80mers. We find that three designs have detectable CD20-specific binding, suggesting that the others may have required avidity effects or the rest of the CAR scaffold for target recognition
Seven of the top 10 designs had broad function across proliferation, expansion, cytokine production, and non-specific cell lysis. Four of the top 10 additionally had target-specific lysis. Two of the failed designs were found to lack CAR expression.
To confirm that the proliferating designs had broad T cell function, the top ten performing designs were evaluated as individual constructs for CAR expression, proliferation, expansion, cytokine production, target cell lysis, and additionally as isolated 80mers for CD20 binding.
Team hits rates ranged from 0.6% to a staggering 38.4%!
And surprisingly, over half of the designs failed to proliferate in either condition, indicating that the designs interfered with CAR expression.
