360Science for Spark - Running on a local cluster


Download and install Apache Spark from https://spark.apache.org/.

Download and install Java from http://java.com/en/download/manual.jsp.

Native Libraries

The native library files should be placed in folder, e.g. /opt/matchithub.lib or /usr/local/lib64, that can be synced with all the processing nodes. Execute the following steps:

cd matchithub-spark/lib
sudo cp lib*.so /usr/local/lib64

Use rsync to sync the /usr/local/lib64 folder with all processing nodes in the cluster. Ensure that spark-env.sh contains these lines:


Running Jobs

Each of the sample application folders contains a run.sh example script showing a spark-submit command.

Was this article helpful?
0 out of 0 found this helpful

have a question or not finding what you're looking for?

Submit a ticket to get some help