Case Study: GrayMeta achieves scalable, secure batch processing with HashiCorp Nomad

Backend Batch Processing At Scale With Nomad A Graymeta Case Study

GrayMeta, a company that indexes large file stores and enriches content with metadata extraction and machine learning, needed a better way to process large back-end batch workloads at scale. Their existing queue-based system across multiple VMs had issues with security, poor bin packing for large files, high disk overprovisioning costs, limited operability, and difficulty supporting multitenant SaaS and on-premises deployments. They turned to HashiCorp Nomad as a simple, container-based orchestration layer that could run in cloud and behind corporate firewalls.

HashiCorp helped GrayMeta re-architect its workflow so a lightweight scheduler submits jobs to Nomad, where containerized workers run with narrowly scoped API/OAuth access and environment-based configuration. This improved isolation and security, reduced operational overhead, enabled better resource utilization and shared clusters across customers, and removed the need to build custom scheduling/bin-packing logic. GrayMeta also gained flexibility to use different schedulers in different environments, including ECS in AWS, while keeping a consistent developer experience with Nomad running locally.

GrayMeta

Jason Hancock

Software and Systems Engineer