Unstructured Image Processing in Real Time
- Ingesting 100,000 images per second across 10 parallel streams,
- creating business value by capturing Satellite Images of weather, hashing them for correlation analysis.
(Circa 2001, still in use today)
Turning unstructured data into information in real-time!
Raytheon had a number of challenges. Their job was to ingest high quality images from satellite feeds (in orbit), correlate the images in real-time, tag them, classify them, and read the important bits.
From there, they were to consolidate and integrate the image results where it made sense. They leveraged a team of 7 people and in the course of 2 months built a fully enabled Data Vault 2.0 solution that captured 100k images per second across 10 parallel streams.
While that may be slow by todays’ standards, in 2001 that was bleeding edge performance. They leveraged a proprietary hash algorithm to key the images, tag and classify them. From there they ran standard analytics – when the hash keys overlapped, then the images were correlated and pulled together for further analysis.
They did not leverage a landing zone, as it was “too slow” for their purposes. By leveraging Data Vault 2.0 model, methodology, and implementation best practices they were able to achieve the results their customers wanted, in record time!