Case Study: Advanced Energy United reduces crawl time from 168 hours to under 4 hours with Arbisoft

Advanced Energy United cuts crawl time from 168 hours to under 4 hours with Arbisoft

Advanced Energy United (AEU) needed to overcome a major data aggregation challenge for its InsightEngine product. The customer required daily crawls of 50 public Utility Commission websites, a process that was taking over a week (168 hours) and incurring significant overhead costs. To address this inefficiency, AEU turned to the vendor, Arbisoft, for its web scraping services.

Arbisoft implemented a custom solution featuring a distributed crawling mechanism and intelligent data extractors capable of semantic analysis. This system, which included real-time data cleaning and a high-performance REST API, drastically reduced the data collection time. The result was a 97.6% decrease in runtime, bringing the crawl time down to less than 4 hours. This improvement enabled AEU to provide subscribers with timely information and achieve substantial cost savings.

Advanced Energy United (aeu)