hadoop-spark

by dancrystalbeach
4 deployments · 3 still active · last rev. 8 months ago

Harden server and setup for Hadoop and Spark

Compatible with: Ubuntu 16.04 LTS, Ubuntu 18.04 LTS, Ubuntu 18.10
						#!/bin/bash
apt-get update && apt-get upgrade -y
apt-get install ufw
ufw allow ssh
ufw enable

apt-get update
apt-get install python2.7 python-pip
apt-get install openjdk-8-jdk
apt-get install wget
apt-get install vim

wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz
tar -xzf hadoop-2.7.3.tar.gz
mv hadoop-2.7.3 hadoop
echo 'PATH=/home/hadoop/hadoop/bin:/home/hadoop/hadoop/sbin:$PATH' >> hadoop/.profile

wget https://archive.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz
tar xvzf spark-2.3.0-bin-hadoop2.7.tgz
ln -s spark-2.3.0-bin-hadoop2.7 spark
echo 'SPARK_HOME=/LinuxHint/spark' >> ~/.bashrc
echo 'export PATH=$SPARK_HOME/bin:$PATH' >> ~/.bashrc