mHUB - 360Science for Spark - DedupeTextFile


The DedupeTextFile application demonstrates using the matchIT Hub for Spark ‘String’ classes to work with text files, delimited record strings, and Spark RDDs.



The command line argument to DedupeTextFile is the name of a configuration file. This is an xml file in the following format:

<?xml version="1.0" encoding="utf-8" ?>

<input table="0" columns="|UniqueRef|FullName|Company|Address1|Address2|City|State|Zip" />

The <dedupeTextFile> section is specific to this application.

mainFile The file to deduplicate, or the main file for an overlap.
overlapFile The file to overlap with the main file. Do not specify this to perform an internal dedupe of a single file.
delimiter The delimiter used in the file and used in the Strings in the RDDs.
outputPath The output folder for the matching output.
licenceFile A file containing the product activation code.

The <hub> section configures the underlying matching engine. Refer to the matchIT Hub documentation for details. The <hub> section must contain the following sub-sections: data, matching, threads, advanced.

Running the sample

cd samples/DedupeTextFile
chmod 777 DedupeTextFile-jar-with-dependencies.jar