CDH(Cloudera Distributed Hadoop) is Cloudera’s open source platform, is the most popular distribution of Hadoop and related projects in the world (with support available via a Cloudera Enterprise subscription)

CDH integrates Hadoop with more than a dozen other critical open source projects. Cloudera has created a functionally advanced system that helps end-to-end Bigdata workflows.

Hadoop Basics:

The Hadoop platform was designed to solve problems where huge amount of data need to processed. It is for the situations where complex analytical operations need to be carried out on the Bigdata. This data may contain structured, unstructured or semi structured in nature.

 Unstructured data comes from various sources and takes many forms like web logs, text files, sensor readings, user-generated content like product reviews or text messages, audio, video and many more.

Dealing with Bigdata requires two things:

1. Cost effective reliable storage and
2. Tools for processing and analyzing unstructured/structured data.

Tools for processing and analyzing unstructured/structured data.

Apache Hadoop is a powerful open source software platform that addresses both of these aspects. Hadoop is an Apache Software Foundation project. Cloudera offers commercial support and services to Hadoop users.

In this Article I am going to explain how to install CDH6 on 3 virtual machine instances with SUSE Linux Enterprise Server 12 SP4 created on Google Cloud Platform in offline mode.

Ways to Install CDH6:

One can install CDH6 in any of the following ways:

1. Installing using Cloudera quickstart VM.
2. Automated method using Cloudera Manager software.
3. Manual methods using native package management tools like yum, zypper, apt.

In this article I am going to explain the 2^nd method.

Pre-requisites:

Below are the pre-requisites before starting the installation

1. Active subscription to Google Cloud Platform(GCP).
2. You should be having access to create VM instances in GCP
3. You should be having root/sudo access to the VM instances created on GCP

Installation Process:

Installation process can be broadly classified into 5 steps and they are

1. Environment setup
2. Local repository setup
3. Cloudera Manager installation
4. Installing CDH through Cloudera Manager
5. Configuring the services

Let us look at the each step in detail.

1. Environment Setup

We will be creating 3 virtual machine instances to install CDH in GCP. Below are the steps

A. Creating VMs in GCP
B. Creating Cluster Using the VMs
C. Creating User with sudo (no password) Access
D. Creating Passwordless SSH
E. Disabling the Firewall

Let us see each step in detail

A. Creating VMs in GCP

Follow the below steps to create VMs in GCP

i. Before creating VMs, we need to create a project in GCP. Access the below URL in your favorite browser to create one

https://console.cloud.google.com/home

ii. Click on the Create button

iii. Provide meaning full name for the project and click on Create

iv. Navigate to Navigation Menu –> Compute Engine –> VM Instances.
To create VM instance, click on Create.

v. Provide name of machine as “earth”, Region as asia-south1(Mumbai) and 2 vCPUs as cores. You need to select Machine type as per your requirement. You can also adjust the default memory allocated to vCPUs.

vi. Under Boot Disk, click on Change to select SUSE Linux Enterprise Server 12 SP4 For SAP x86_64 OS Image. Update the size of the disk to 20GB. On disk size you need to take call as per requirement.

vii. Select Allow HTTP Traffic and Allow HTTPS Traffic options and click on Create to start creating the VM.

viii. Similarly create another 2 VMs with names mercury and venus. Once all three VMs are created, VM instances page would look like as follows. Click on the highlighted Play button to spin up the machines if they are not started already.

Now the 3 VMs with names earth, mercury and venus are create. Each VM is assigned with an internal IP and an external IP. Internal IPs are used to communicate within the project where as external IPs are required to access the machines from different network.

B. Creating Cluster Using the VMs

To create cluster out of the VMs which we have just created, please follow below steps.

i. In order to set root credential click on SSH under Connect column for earth VM. This will start the shell with your user ID as shown in the below screenshot. Impersonate as root with “sudo su” command and set the root password with “passwd” command

ii. Execute below commands to enable SSH through public key. As a side note these command should be run as root or user with sudo access. These commands should be run on all the there machines.

sudo sed -i 's/#RSAAuthentication\ yes/RSAAuthentication\ yes/g' /etc/ssh/sshd_config
sudo sed -i 's/#PubkeyAuthentication\ yes/PubkeyAuthentication\ yes/g' /etc/ssh/sshd_config
sudo sed -i 's/PasswordAuthentication\ no/PasswordAuthentication\ yes/g' /etc/ssh/sshd_config
sudo systemctl restart sshd

iii. Add Internal IP addresses of all three machines along with their fully qualified hostnames and short names to /etc/hosts file. These information can be obtained with below command. As a side note root user or user with sudo access can update this file.

Attribute	Command
Internal IP Address	ip address
Fully Qualified Hostname	hostname -f
Short Hame of Hostname	hostname

Once updated the file should look like as follows

Same information should also be updated in the /etc/hosts file of mercury and venus VMs

iv. Update the /etc/hostname file with fully qualified hostname

C. Creating User with sudo (no password) Access

To create user and provide necessary access follow the below steps. These steps need to be carried out on all the 3 VMs.

i. Create user cdh and set the password with below commands. These commands need to be executed as root or user with sudo access.

sudo useradd -m cdh
sudo passwd cdh

ii. Now execute the below command as root or user with sudo access to provide sudo(no password) permission to the cdh user.

sudo echo "cdh ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

D. Creating Passwordless SSH

We need to create passwordless ssh from earth VM to mercury, venus and earth VM itself. Please follow the below steps to do the same

i. In earth VM, if you have already logged in as root user, impersonate to cdh user as follows

su - cdh

ii. With below command create ssh key. You can also provide optional pass phrase and location for the ssh key

ssh-keygen

iii. Copy the generate ssh key to earth, mercury and venus VMs with below command. While copying it will ask password of the cdh user and that is one time activity.

ssh-copy-id –i ~/.ssh/id_rsa cdh@venus.asia-south1-b.c.savvy-serenity-225906.internal
ssh-copy-id –i ~/.ssh/id_rsa cdh@mercury.asia-south1-b.c.savvy-serenity-225906.internal
ssh-copy-id –i ~/.ssh/id_rsa cdh@earth.asia-south1-b.c.savvy-serenity-225906.internal

D. Disabling the Firewall

To disable the firewall please execute below commands.

SuSEFirewall2 off
SuSEFirewall2 stop

2. Local Repository Creation

I am going install apache2 web server to create local repository. This process involves in 2 parts

A. Installing the web server
B. Downloading the Cloudera repository to local environment

A. Web Server Installation

i. Run the below command to install apache2 webserver

sudo zypper
in apache2

ii. Set directory “Option” to All from None in /etc/apache2/default-server.conf file

iii. Start the apache2 server using below command

systemctl start apache2

B. Cloudera Local Repository Creation:

Run the below set of commands to create local cloudera repository

sudo mkdir -p /srv/www/htdocs/cloudera-repos
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cm6/6.0.0/sles12/ -P /srv/www/htdocs/cloudera-repos
sudo chmod -R ugo+rX /srv/www/htdocs/cloudera-repos/cm6
cd /srv/www/htdocs/cloudera-repos/cm6/6.0.0
sudo wget https://archive.cloudera.com/cm6/6.0.0/allkeys.asc

To make sure every thing is working fine access the apache webserver through web browser. To do that we will be required external/public IP of the machine. This can be obtained from VM instances page of GCP. Access the IP through your favorite web browser. You should be able to see something like this

3. Clouder Manager Installation

Before starting cloudera manager we need to install the dependency on all the machines. Below command installs PostgreSQL adapter for python. Since we will be using default PostgreSQL database to maintain metadata it is required. This need to executed in all the 3 machines. As a side note, for better readability I have entered below command in multiple lines with the help of \ character.

sudo zypper addrepo https://download.opensuse.org/repositories/server:database:postgresql/\
openSUSE_Tumbleweed/server:database:postgresql.repo \
&& sudo zypper refresh && sudo zypper install python-psycopg2

Cloudera provides cloudera-manager-installer.bin file to install cloudera manager. Below are the installation steps for the same

i. Create ssh connection to earth machine as cdh user and download cloudera-manager-installer.bin file as follows

wget https://archive.cloudera.com/cm6/6.0.0/cloudera-manager-installer.bin 

ii. Provide executable permission to the downloaded file as follows

chmod uga+x
cloudera-manager-installer.bin

iii. Run the file with sudo permission

sudo ./cloudera-manager-installer.bin

iv. Installation wizard pops up as follows. Read the information and proceed accordingly. Click Next to continue

v. Go through the license, if you are agreeing to the terms and condition click Next

vi. Click on Yes if you agree to the license terms and conditions

vii. Next the wizard pops up Oracle Binary Code License Agreement for the Java SE. Go through the License terms and condition carefully. If you agree to that click on Next

viii. If you are agreeing to license terms and conditions click on Yes

ix. Once installation is completed below screen will appear. Access the URL in web browser. Since the machine is remote, I am going to use its external IP instead of localhost.

x. By default the 7180 port would not be accessible. We need to create a firewall rule to open the port for the machine from which we are going to accessing the URL. This can be done as follows. Navigate to Google Cloud Platform –> Compute Engine –> VM Instancees and click on nic0 corresponding to earth.

xi. Click on Firewall Rules

xii. To create Firewall rule, click on Create Firewall Rule

xiii. Provide all the necessary information as follows
Name: cm
Targets: All instance in the network
Source IP Ranges:
Select tcp checkbox and value for tcp would be 7180.
After providing all these information click on Create.

Note: IP of the local machine can be obtained by searching “my ip address” in your favorite search engine

xiv. When we access the URL like http://<external-ip>:7180 below screen will appear. In my case the URL was like http://35.200.218.170:7180

Default administrative credentials are
Username: admin
Password: admin

xv. After login, wizard oriented installation steps will appear as follows. Click on Continue to proceed.

xvi. Go through the Cloudera Standard license terms and conditions, if you are agreeing select the check box and click on Next.

xvii. I am going with default edition and click on Continue.

xviii. Another installation wizard will start as follows. Click on Continue

xix. Cloudera provides Auto TLS setup steps. Once this is done click on Continue

xx. In the hostname text box, provide fully qualified hostname of all the machines as shown in the below screenshot. Click on Search. Once all the machines are resolved to their internal IP click on Continue.

xxi. Select Custom Repository and provide the custom repository URL as shown in the screenshot and click on Continue.

xxii. Select “Install Oracle Java SE Development Kit(JDK 8)” and click on Continue

xxiii. Select “Another User” and provided the newly created user “cdh”. Enter its password and confirm it by retyping the respective text box and click Continue.

xxiv. Installation on all machine will progress as follows. Once completed successfully click on Continue.

xxv. Download, distribution, unpacking, and activation occur one after the other. Once completed click on Continue.

xxvi. The installation process will Inspects Hosts for correctness. Once completed and there is no error click on Finish.

xxvii. Depending on your need select the option. I am going with All Services

xxviii. Now the installation wizard allows to configure different services on different nodes. I have selected them as follows.

xxix. Perform the Test Connection of database of different services. Post success click on Continue.

xxx. Wizard will take you to the configuration page. I am going with the default configuration and click on Continue.

xxxi. Cloudera Manager will start the services first time. Post successful completion click on Continue

xxxii. Click on Finish.

xxxiii. The services along with their status would appear like follows

Conclusion:

This is how Cloudera Distributed Hadoop can be installed through Cloudera Manager and downloaded parcels on SUSE 12 operating system. This is very useful when the installation need to be done on the cluster which is secure and where internet access is not available. However, you may run into issues due to unanticipated reason as every environment in Linux world is different. Please visit and follow our community site to post the errors you encountered.

Reference:

www.cloudera.com

https://www.cloudera.com/downloads/manager/6-0-0.html

https://www.cloudera.com/documentation/enterprise/6/latest/topics/installation.html