Convert snapshot feeds to audits using J2SE supported
Digital Signatures
By Ramkartik Mulukutla
In this article I examine the usage of digital signatures to convert snapshot loads to audits
(or deltas). Large scale data warehouses and enterprise applications receive snapshot
(data in its entirety each time) data feeds from external systems like ERP and main
frames. These data feeds can come in via a protocol like SOAP or either plain CSV
(Comma separated values) files or in some cases database to database replication. I
discuss detailed techniques that make use of J2SE (Java 2 Standard Edition) supported
digital signatures to solve the problem of weeding out duplicates in incoming data and
effectively converting the snapshots to audits at the receiving end.
The incoming data is likely to have repetitions from previous runs as legacy systems that
deliver such data most likely do not have the capability to give accurate delta changes (or
changes that occurred since the last run). The receiving end which is typically a web
application will have to bear the burden of ensuring that the incoming data is cleaned,
duplicates removed and only true deltas are loaded.
In this article I take a common example of Part-Master data load. Supply chain web
applications, Order management systems, Inventory management systems typically
receive part master data on a periodic basis. This data consists of details of Parts such as
part description, manufacturer, primary supplier, alternates, dimensions, unit of measure,
and in some cases sub part data (bill-of-material). The number of parts can be in hundreds
of thousands depending on the industry. Most of the parts may never change in any of
their attributes during their lifetime. The challenge is to scan through the large volume of
part data and mine those parts that have changed, and load the changed parts. Commonly
the parts that have changed will be inserted into the target system as new versions of an
existing part.
The concepts described here are