-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathatom.xml
More file actions
108 lines (94 loc) · 13.1 KB
/
Copy pathatom.xml
File metadata and controls
108 lines (94 loc) · 13.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>digitalshell</title>
<link href="/atom.xml" rel="self"/>
<link href="http://digitalshell.xyz/"/>
<updated>2017-08-02T12:17:55.000Z</updated>
<id>http://digitalshell.xyz/</id>
<author>
<name>digitalshell</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>path B 安装cloudera cdh</title>
<link href="http://digitalshell.xyz/2017/08/02/path-B-%E5%AE%89%E8%A3%85cloudera-cdh/"/>
<id>http://digitalshell.xyz/2017/08/02/path-B-安装cloudera-cdh/</id>
<published>2017-08-02T11:26:55.000Z</published>
<updated>2017-08-02T12:17:55.000Z</updated>
<content type="html"><![CDATA[<h4 id="安装Cloudera-Manager-and-CDH的几种方式"><a href="#安装Cloudera-Manager-and-CDH的几种方式" class="headerlink" title="安装Cloudera Manager and CDH的几种方式"></a>安装Cloudera Manager and CDH的几种方式</h4><ul>
<li>PATH A： 通过Cloudera Manager自动安装 (不适合生产环境)</li>
<li>PATH B： Installation Using Cloudera Manager Parcels or Packages</li>
<li>PATH C： Manual Installation Using Cloudera Manager Tarballs</li>
</ul>
<h4 id="服务器环境准备"><a href="#服务器环境准备" class="headerlink" title="服务器环境准备"></a>服务器环境准备</h4><ul>
<li><p>SSH无密码验证配置</p>
<ol>
<li>任一台主机(A)执行<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">ssh-keygen</div><div class="line">cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</div><div class="line">chmod 600 ~/.ssh/authorized_keys</div></pre></td></tr></table></figure>
</li>
</ol>
<a id="more"></a>
<ol>
<li><p>使用ssh-copy-id命令将公钥传送到远程主机(B)上</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">ssh-copy-id root@B</div></pre></td></tr></table></figure>
<p>这样从A就可以无密码登录B</p>
</li>
</ol>
</li>
<li>关闭selinux</li>
<li>配置hosts</li>
<li>配置hostname</li>
<li>关闭防火墙并设置开机不启动<ol>
<li>重启后永久性生效：<br>开启：<code>chkconfig iptables on</code><br>关闭：<code>chkconfig iptables off</code></li>
<li>即时生效，重启后失效：<br>开启：<code>service iptables start</code><br>关闭：<code>service iptables stop</code></li>
</ol>
</li>
<li>优化虚拟内存需求率<ol>
<li>检查虚拟内存需求率 <code>cat /proc/sys/vm/swappiness</code></li>
<li>临时修改 <code>sysctl vm.swappiness=10</code></li>
<li>永久修改 <code>vi /etc/sysctl.conf</code> <code>vm.swappiness = 10</code></li>
</ol>
</li>
<li>解决透明大页面问题<ol>
<li>检查 <code>cat /sys/kernel/mm/transparent_hugepage/defrag</code></li>
<li>临时修改<br><code>echo never > /sys/kernel/mm/transparent_hugepage/defrag</code><br><code>echo never > /sys/kernel/mm/transparent_hugepage/enabled</code></li>
<li>永久修改 <code>vi /etc/rc.local</code> 加入上述语句</li>
</ol>
</li>
</ul>
<h4 id="数据库安装配置："><a href="#数据库安装配置：" class="headerlink" title="数据库安装配置："></a>数据库安装配置：</h4><ol>
<li>安装MySQL数据库</li>
<li>mysql-connector<br> <code>mkdir -p /usr/share/java/</code><br> <code>cd /usr/share/java/</code><br> <code>wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz</code><br> <code>tar -zxvf mysql-connector-java-5.1.41.tar.gz</code><br> <code>cp mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar /usr/share/java/mysql-connector-java.jar</code></li>
<li>为Cloudera Manager Server初始化所需的数据库<br> <code>grant all on *.* to 'temp'@'%' identified by 'temp' with grant option;</code><br> <code>/usr/share/cmf/schema/scm_prepare_database.sh mysql -h 192.168.100.222 -utemp -ptemp --scm-host 192.168.100.222 scm scm scm</code><br> <code>drop user 'temp'@'%';</code><br> <a href="https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_installing_configuring_dbs.html#concept_mff_xjm_hn" target="_blank" rel="external">https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_installing_configuring_dbs.html#concept_mff_xjm_hn</a></li>
<li>创建hue,hive等其他服务所需的数据和账号 <figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div></pre></td><td class="code"><pre><div class="line">grant all privileges on *.* to 'scm'@'%' identified by 'scm' WITH GRANT OPTION;</div><div class="line">GRANT ALL PRIVILEGES ON *.* TO 'scm'@'%' IDENTIFIED BY 'scm' WITH GRANT OPTION;</div><div class="line">GRANT ALL PRIVILEGES ON *.* TO 'scm'@'192.168.100.235' IDENTIFIED BY 'scm' WITH GRANT OPTION;</div><div class="line"></div><div class="line">GRANT ALL ON *.* to 'root'@'%' identified by '123456' WITH GRANT OPTION;</div><div class="line"></div><div class="line">mkdir -p /usr/share/java/</div><div class="line">wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz</div><div class="line">tar -zxvf mysql-connector-java-5.1.41.tar.gz</div><div class="line">cp mysql-connector-java-5.1.41/mysql-connector-java-5.1.41.39-bin.jar /usr/share/java/mysql-connector-java.jar</div><div class="line"></div><div class="line"></div><div class="line">/usr/share/cmf/schema/scm_prepare_database.sh mysql -h 192.168.100.222 -utemp -ptemp --scm-host 192.168.100.222 scm scm scm</div><div class="line"></div><div class="line">create database amon DEFAULT CHARACTER SET utf8;</div><div class="line">grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';</div><div class="line">grant all privileges on amon.* TO 'amon'@'192.168.100.222' IDENTIFIED BY 'amon_password';</div><div class="line"></div><div class="line">create database rman DEFAULT CHARACTER SET utf8;</div><div class="line">grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';</div><div class="line"></div><div class="line">create database metastore DEFAULT CHARACTER SET utf8;</div><div class="line">grant all on metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';</div><div class="line">grant all on metastore.* TO 'hive'@'master.hadoop.dev.8dol.com' IDENTIFIED BY 'hive_password';</div><div class="line"></div><div class="line">create database sentry DEFAULT CHARACTER SET utf8;</div><div class="line">grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry_password';</div><div class="line"></div><div class="line">create database nav DEFAULT CHARACTER SET utf8;</div><div class="line">grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav_password';</div><div class="line"></div><div class="line">create database navms DEFAULT CHARACTER SET utf8;</div><div class="line">grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms_password';</div><div class="line"></div><div class="line">create database oozie;</div><div class="line">grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';</div><div class="line">grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';</div><div class="line">grant all privileges on oozie.* TO 'oozie'@'192.168.100.222' IDENTIFIED BY 'oozie';</div><div class="line"></div><div class="line">create database hue;</div><div class="line">grant all privileges on hue.* to 'hue'@'localhost' identified by 'hue';</div><div class="line">grant all privileges on hue.* to 'hue'@'%' identified by 'hue';</div><div class="line">grant all privileges on hue.* TO 'hue'@'192.168.100.222' IDENTIFIED BY 'hue';</div></pre></td></tr></table></figure>
</li>
</ol>
<p>参考文档：<br><a href="https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_mysql.html#cmig_topic_5_5_2" target="_blank" rel="external">https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_mysql.html#cmig_topic_5_5_2</a></p>
<h4 id="手动通过Cloudera-Manager安装CDH（PATH-B）"><a href="#手动通过Cloudera-Manager安装CDH（PATH-B）" class="headerlink" title="手动通过Cloudera Manager安装CDH（PATH B）"></a>手动通过Cloudera Manager安装CDH（PATH B）</h4><ol>
<li>安装JDK</li>
<li>安装数据库</li>
<li>安装启动Cloudera Manager Server</li>
<li>安装Cloudera Manager Agents（手动或通过Cloudera Manager Installation wizard）</li>
<li>安装CDH and Managed Service software（手动或通过Cloudera Manager Installation wizard）</li>
<li>创建、配置、启动CDH and Managed Services<br>参考文档：<br><a href="https://www.cloudera.com/documentation/enterprise/latest/topics/installation_installation.html" target="_blank" rel="external">https://www.cloudera.com/documentation/enterprise/latest/topics/installation_installation.html</a><br><a href="https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html#cmig_topic_6_6_1" target="_blank" rel="external">https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html#cmig_topic_6_6_1</a></li>
</ol>
<h3 id="错误解决"><a href="#错误解决" class="headerlink" title="错误解决"></a>错误解决</h3><ul>
<li>启动hdfs时，报错 Canary 测试无法在目录 /tmp/.cloudera_health_monitoring_canary_files 中创建文件。<br>经过查看日志，发现 Name node is in safe mode.<br>解决方法：sudo -uhdfs hdfs dfsadmin -safemode leave</li>
</ul>
<p>参考文档：<br><a href="http://www.jianshu.com/p/57179e03795f" target="_blank" rel="external">http://www.jianshu.com/p/57179e03795f</a><br><a href="http://cmdschool.blog.51cto.com/2420395/1775398" target="_blank" rel="external">http://cmdschool.blog.51cto.com/2420395/1775398</a><br><a href="http://blog.csdn.net/wendingzhulu/article/details/52423580" target="_blank" rel="external">http://blog.csdn.net/wendingzhulu/article/details/52423580</a></p>
]]></content>
<summary type="html">
<h4 id="安装Cloudera-Manager-and-CDH的几种方式"><a href="#安装Cloudera-Manager-and-CDH的几种方式" class="headerlink" title="安装Cloudera Manager and CDH的几种方式"></a>安装Cloudera Manager and CDH的几种方式</h4><ul>
<li>PATH A： 通过Cloudera Manager自动安装 (不适合生产环境)</li>
<li>PATH B： Installation Using Cloudera Manager Parcels or Packages</li>
<li>PATH C： Manual Installation Using Cloudera Manager Tarballs</li>
</ul>
<h4 id="服务器环境准备"><a href="#服务器环境准备" class="headerlink" title="服务器环境准备"></a>服务器环境准备</h4><ul>
<li><p>SSH无密码验证配置</p>
<ol>
<li>任一台主机(A)执行<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">ssh-keygen</div><div class="line">cat ~/.ssh/id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys</div><div class="line">chmod 600 ~/.ssh/authorized_keys</div></pre></td></tr></table></figure>
</li>
</ol>
</summary>
<category term="大数据" scheme="http://digitalshell.xyz/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
</entry>
</feed>