Case Study: Recursion Pharmaceuticals accelerates drug discovery with Confluent Cloud

Recursion Accelerates Drug Discovery with New Data Pipelines Based on Confluent Cloud

Recursion Pharmaceuticals faced growing bottlenecks as its high-throughput screening lab generated massive biological image data—more than three petabytes and single experiments over 8 TB—forcing slow batch processing and preventing real-time quality control. To solve this, Recursion turned to Confluent, adopting Confluent Cloud and Apache Kafka (with Kafka Streams) to enable event streaming and to reuse existing microservices with lower operational overhead.

Using Confluent Cloud, Kafka Streams and a Dagger orchestration layer, Recursion built a scalable, highly available event-driven pipeline that uploads images to Google Cloud, autos-scales workers on GKE, and processes images and metrics in real time; Confluent provided the persistent log model and simplified operations. The result: feature extraction that took about an hour on the old batch system can now scale to screen thousands of compounds against hundreds of disease models in minutes, the pipeline has been stable in production for over a year, and Recursion reports concrete progress—30+ disease models in discovery, nine in preclinical development and two in clinical trials.

Recursion Pharmaceuticals

Ben Mabey

VP of Engineering