Case Study: HG Data achieves rapid, scalable big-data processing and 50% cost savings with Amazon Web Services

HG Data - Customer Case Study

HG Data is a leader in competitive intelligence, indexing billions of unstructured documents to create a detailed global census of B2B technology installations used for market analysis, predictive modeling and sales. After selling their previous business and losing access to co-location infrastructure, the founders needed a fast, scalable replacement to process massive batch and crawl data (roughly 10 TB/month and about a billion documents per quarter) and to eliminate long hardware lead times and operational bottlenecks.

HG Data migrated to AWS, using S3 for storage, Amazon EMR/Hadoop for large-scale deduplication and processing, MongoDB/Elasticsearch/RDS for indexing and delivery, and EC2 (including automated Spot and Reserved instances) with custom APIs to automate workflows. The move enabled linear-scale processing of 10+ billion documents, cut infrastructure costs (about 50% lower CAPEX and up to 70% savings with Spot bidding), and shrank delivery timeframes from weeks to hours or days—allowing rapid, on-demand customer delivery.

HG Data

Craig Harris

Co-founder and CEO