Hadoop and Spark overview

Overview


Apache Hadoop is a collection of open source cluster computing tools that supports popular applications for data science at scale, such as Spark.

You can interact with Hadoop from your Domino executors by configuring your Domino environment with the necessary software dependencies and credentials. Domino supports most providers of Hadoop solutions, including MapR, Cloudera, and Amazon EMR. Once a Domino environment is set up to connect to your cluster, Domino projects can use the environment to work with Hadoop applications.

 

Video introduction to using Spark in Domino

For a thorough video introduction to using Spark in Domino, watch the webinar recording:

 

 

 

Using a Hadoop-enabled environment in your Domino project


If your Domino administrators have already created an environment for connecting to a Hadoop cluster, you can follow these subsections of the setup instructions to use that environment in your Domino project.

 

For users setting up projects to work with an existing environment, read these subsections:

 

Once your project is set up to use the environment, you can execute code in your Domino Runs that connects to the cluster for Spark, HDFS, or Hive functionality.

 

Screen_Shot_2019-04-26_at_10.23.17_AM.png

 

 

 

Setting up Domino to connect to a new Hadoop cluster


To connect to your existing Hadoop cluster from Domino, you must create a Domino environment with the necessary dependencies installed. Some of these dependencies, including binaries and configuration files, will come directly from the cluster itself. Others will be external software dependencies like Java and Spark, and you will need to match the version you install in the environment to the version running on the cluster.

The basic steps for setting up an environment to connect to your cluster are:

  1. Gather binaries and configuration files from your cluster
  2. Gather dependencies from external sources, like Java JDKs and Spark binaries
  3. Upload all dependencies to a Domino project, to make them accessible to the Domino environment builder
  4. Author a new Domino environment that pulls from the Domino project, then installs and configures all required dependencies

 

Screen_Shot_2019-04-26_at_10.29.18_AM.png

 

For Domino admins setting up a Domino environment to connect to a new cluster, read the full provider-specific setup guides:

 

 

 

Additional capabilities


Domino also supports running Spark on a Domino executor in local mode, querying Hive tables with JDBC, and authenticating to clusters with Kerberos. See the following guides for more information.

Was this article helpful?
0 out of 0 found this helpful