离线安装CDH

  |   0 评论   |   1,236 浏览

安装调研

准备安装最新的CDH 5.14.2,其中各包的版本见官方文档

Component Package Version Tarball Release Notes Changes File
Apache Avro avro-1.7.6+cdh5.14.2+137 Tarball Release notes Changes
Apache Crunch crunch-0.11.0+cdh5.14.2+104 Tarball Release notes Changes
Datafu pig-udf-datafu-1.1.0+cdh5.14.2+26 Tarball Release notes Changes
Flume-ng flume-ng-1.6.0+cdh5.14.2+181 Tarball Release notes Changes
Apache Hadoop hadoop-2.6.0+cdh5.14.2+2748 Tarball Release notes Changes
Hadoop Mrv1 hadoop-0.20-mapreduce-2.6.0+cdh5.14.2+2748 (none) (none) (none)
Hbase hbase-1.2.0+cdh5.14.2+456 Tarball Release notes Changes
Hbase-solr hbase-solr-1.5+cdh5.14.2+73 Tarball Release notes Changes
Apache Hive hive-1.1.0+cdh5.14.2+1344 Tarball Release notes Changes
Hue hue-3.9.0+cdh5.14.2+7875 Tarball Release notes Changes
Apache Impala impala-2.11.0+cdh5.14.2+0 (none) Release notes Changes
Kite SDK kite-1.0.0+cdh5.14.2+146 Tarball Release notes Changes
Apache Kudu kudu-1.6.0+cdh5.14.2+0 (none) Release notes Changes
Llama llama-1.0.0+cdh5.14.2+0 Tarball Release notes Changes
Apache Mahout mahout-0.9+cdh5.14.2+35 Tarball Release notes Changes
Apache Oozie oozie-4.1.0+cdh5.14.2+484 Tarball Release notes Changes
Apache Parquet parquet-1.5.0+cdh5.14.2+196 Tarball Release notes Changes
Parquet-format parquet-format-2.1.0+cdh5.14.2+19 Tarball Release notes Changes
Apache Pig pig-0.12.0+cdh5.14.2+113 Tarball Release notes Changes
Cloudera Search search-1.0.0+cdh5.14.2+0 Tarball Release notes Changes
Apache Sentry sentry-1.5.1+cdh5.14.2+444 Tarball Release notes Changes
Apache Solr solr-4.10.3+cdh5.14.2+523 Tarball Release notes Changes
Apache Spark spark-1.6.0+cdh5.14.2+543 Tarball Release notes Changes
Apache Sqoop sqoop-1.4.6+cdh5.14.2+130 Tarball Release notes Changes
Apache Sqoop2 sqoop2-1.99.5+cdh5.14.2+47 Tarball Release notes Changes
Apache Whirr whirr-0.9.0+cdh5.14.2+24 Tarball Release notes Changes
Apache Zookeeper zookeeper-3.4.5+cdh5.14.2+142 Tarball Release notes Changes

下载安装包

manager文件

http://archive.cloudera.com/cm5/cm/5/

或者使用国内镜像下载

http://cloudera.proxy.ustclug.org/cm5/cm/5

parcel文件

http://archive.cloudera.com/cdh5/parcels/latest/

在下载对应的安装文件,我这里为:

同样可以使用国内镜像下载

http://cloudera.proxy.ustclug.org/cdh5/parcels/latest/

环境准备

配置hosts文件

目标:在每一台机器上,都能ping通各台机器的主机名。

我这里有DNS解析,就不配置了。

ssh 免登录

略. 可参考 离线安装Cloudera Manager 5和CDH5(最新版5.1.3) 完全教程

安装Oracle JDK

略. 可参考 离线安装Cloudera Manager 5和CDH5(最新版5.1.3) 完全教程

安装MySQL

建表

#hive
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

#activity monitor
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

允许root远程登录

grant all privileges on *.* to 'root'@'%' identified by 'helloworld' with grant option;

flush privileges;

关闭防火墙和SELinux

所有节点配置NTP服务

安装Cloudera Manager

主节点解压安装

在主节点上新建目录/home/admin/hadoop,解压cloudera-manager-el6-cm5.14.2_x86_64.tar.gz到此目录.

为Cloudera Manager 5建立数据库

下载mysql驱动

wget "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.zip"

解压,将其中的jar文件放在目录/home/admin/hadoop/cm-5.14.2/share/cmf/lib/

在主节点初始化CM5的数据库:

usage: /home/admin/hadoop/cm-5.14.2/share/cmf/schema/scm_prepare_database.sh [options] (postgresql|mysql|oracle) database username [password]
/home/admin/hadoop/cm-5.14.2/share/cmf/schema/scm_prepare_database.sh mysql cm -hmysqlhost -uroot -pxxx --scm-host localip scm scm scm

或者 5.14.3

/home/admin/hadoop/cm-5.14.3/share/cmf/schema/scm_prepare_database.sh mysql cm root password

执行结果

[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

Agent配置

同步Agent到其他节点

scp /home/admin/hadoop/cm-5.14.2.tar.gz $i:/home/admin/

启动服务端

修改/home/admin/hadoop/cm-5.14.2/etc/init.d/cloudera-scm-server中的用户名cloudera-scm为自己运行的用户名

USER=cloudera-scm
GROUP=cloudera-scm

启动服务端

/home/admin/hadoop/cm-5.14.2/etc/init.d/cloudera-scm-server start

启动agent端

/home/admin/hadoop/cm-5.14.2/etc/init.d/cloudera-scm-agent start

我这里启动agent失败了,查看日志/home/admin/hadoop/cm-5.14.2/log/cloudera-scm-agent/cloudera-scm-agent.log知道是9000端口被占用了

修改配置文件/home/admin/hadoop/cm-5.14.2/etc/cloudera-scm-agent/config.ini中的端口

listening_port=9000

然后再次启动成功.

Q: 异常 MainThread agent ERROR Heartbeating to localhost:7182 failed.
A: 修改配置文件:/home/admin/hadoop/cm-5.14.2/etc/cloudera-scm-agent/config.ini中的server_host

Q: 异常Permission denied: '/opt/cloudera'
A: 修改配置文件:/home/admin/hadoop/cm-5.14.2/etc/cloudera-scm-agent/config.ini中的/home/admin/hadoop/cloudera/parcels

Q: 主机运行状态不良
A: rm -f /opt/cloudera-manager/cm-5.11.1/lib/cloudera-scm-agent/cm_guid, 再重启agent

安装hadoop

Cloudera Manager Server和Agent都启动以后,就可以进行CDH5的安装配置了。

通过浏览器访问主节点的7180端口,默认的用户名和密码均为admin

Q: 找不到MySQL驱动
A: 脚本中会从/usr/share/java/mysql-connector-java.jar来复制mysql驱动,所以自己先做个软链

ln -sf /home/admin/hadoop/cm-5.14.2/share/cmf/lib/mysql-connector-java-5.1.46.jar /usr/share/java/mysql-connector-java.jar

Q: Hue启动失败
A: 需要安装httpdmod_ssl依赖包

还有一些坑,不过最终安装成功了.

参考

评论

发表评论

validate