Setting up SSH for a Hadoop cluster
When setting up aHadoop cluster, you’ll need to designate one specific node as the master node.We need to define a means for the master node to remotely access every node inyour cluster. Hadoop uses pasphraseless SSH for this purpose. SSH utilizesstandard
public key cryptography to create a pair of keys for user verification– one public, one private. The public key is stored locally on every node inthe cluster, and the master node sends the private key when attempting toaccess a remote machine. With both pieces
of information, the target machinecan validate the login attempt.
We’ve been speakingin general terms of one node accessing another; more precisely this access isfrom a user account on one node to another user account on the target machine.For Hadoop, the accounts should have the same username on all of the nodes, andfor
security purpose we recommend it being a user-level account. This accountis only for managing your hadoop cluster. Once the Cluster daemons are up andrunning, you’ll be able to run your actual MapReduce jobs from other accounts.
Check whether SSHis installed on your nodes, if not please install OpenSSH via a Linux packagemanager or by downloading the source directly.
$ which ssh
/usr/bin/ssh
$ which ssd
/usr/bin/which: nossd in(/usr/kerberos/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin)
$ which ssh-keygen
/usr/bin/ssh-keygen
Having verifiedthat SSH is correctly installed on all nodes of the cluster, We use ssh-keygenon the master node to generate an RSA key pair. Be certain to avoid entering apassphrase, or you’ll have to manually enter that phrase every time the masternode
attempts to access another node.
$ ssh-keygen -t rsa
Generating 2048-bitrsa key pair
4 .oOo.oOo.oOO
Key generated.
2048-bit rsa,yj70978@retailvm1d, Mon Jul 22 2013 21:06:38 -0400
Passphrase :
Again :
Key is stored withNULL passphrase.
(You can ignore the following warning if youare generating hostkeys.)
This is not recommended.
Don't do this unless you know what you'redoing.
If file system protections fail (someone canaccess the keyfile),
or if the super-user is malicious, your keycan be used without
the deciphering effort.
Private key savedto /home/yj70978/.ssh2/id_rsa_2048_a
Public key saved to/home/yj70978/.ssh2/id_rsa_2048_a.pub
$ ls -rlt/home/yj70978/.ssh2/id_rsa_2048_a
-rw------- 1yj70978 retailfi 1539 Jul 22 21:07 /home/yj70978/.ssh2/id_rsa_2048_a
$ cat /home/yj70978/.ssh2/id_rsa_2048_a
---- BEGIN SSH2ENCRYPTED PRIVATE KEY ----
Subject: yj70978
Comment:"2048-bit rsa, yj70978@retailvm1d, Mon Jul 22 2013 21:06:38 -\
0400"
P2/56wAAA+wAAAA1aWYtbW9kbntzaWdue3JzYS1wa2NzMS1zaGExfSxlbmNyeXB0e3JzYS
1wa2NzMS1ub25lfX0AAAAEbm9uZQAAA58AAAObAAAAEQEAAQAAB/oC01cYvh8cNTVtBELB
azF4B2MT2tpf0MWvA0W2KEho5G3O/TnjOocBRpdRn/yDLJieqinxzBpX9rYumP0/So3FIF
VUUd0TyLEj9CikIjVQWj0KUvNKdjsDzujiWOEpfe0vUMkqosc1GUsdIDXaBAA/LIAuTQjF
dlsfYp/OcwOFyRm4+v3lwJqWT4hedPLhZmqNgfiXOehofe1T2B5W4d/+zJ1Y2BCk9Nrak+
bJGM+Za+v1HfOgxNY+XJCpBdxb0GA9Pt8VvGjSi3liQkM9LEWGk9lxUXtzD2lsaDImmwcD
WhJ/FJVV+CNirFcSKN5Z/a8BhGa0MQFRww6hKECJRet1AAAIAKosmjEkUBCzLjPrFjwitp
zOCF6OsuAACfyyLq6jex7R78mBt0FhuezZXIvkvgxCS1gNoacBU7p0fK5d1Nte3NUC5B5k
yKuvIcEtlX8zMgu8aqA1ZDMETr3+C6qfyBiQ690sxfOEsBnW4T2Yo1KAbylOisFuyMq+dh
35nV2hLzNSVLRpiTkZniMYPQHe7o/Kj0zSQtBJNtTAgyc4JkxThYKHEdJOfZ0Zah9hVrqV
wKq2Rwf4eWmZjDABD9fzAPQOVljyaVlAQf3/m4sK5eAX1QbxPsuEUNJ4p/W/SCPaNyZYZD
UlSoEOYueWkHY/yVD5labPLUgFg3p00pUPEtjt/6sAAAP9GTPVWw+wvnbn8aKIRZ6pJpn+
zurN/c8j7wi0hT+63+UShUXO59/EadI5THTlEyB1vVBF78iuoZZfB6t4s3p4dlk/Ke0CxH
Xn81ZJy6r4kd5VhZv2fEKOjq03aewONRX1K4GXbD23KqwokVgIQ9YsJ9A187hbBz8S1N2e
XqP499wAAAQAwUumdnfKNCrr0+a5kwyEvsW85SqgDnVOeP1On1garxE6O5EVYbrlRpgk+O
kDMj7x23LwT9VnZqFcmEO8cpuo3csUzMAd0cX3hkod6RKZ/H1btrEmpzkvFA2Kf7FX3D5i
uKKq5pWykEM5fZjvXe1NbFKOCXytkqnMKJ120Bm84IUAAAQA4WDUlumoGtM/vOZLge9DyW
jO1/lfr7WCoPCFne5APOVLtGMo/xlBFVmnPrnJ6deq2EMk7IRI9ru9SB1TDNV1ktXUH6af
vR6r6vhQbHytjnd1nDeiS3xODS+ePlGwH8RgWzIU+X3uOgsKNJW7Q2OqC3uGcyIGshhc+b
9lXDop7m8=
---- END SSH2ENCRYPTED PRIVATE KEY ----
Albeit a bit tedious,you’ll next need to copy the public key to every slave node as well as themaster node:
Scp ~/.ssh/d_rsa.pubhadoop-user@target:~/master_key
Manually log in tothe target node and set the master key as an authorized key (or append to thelist of authorized keys if you have others defined).
After generatingthe key, you can verify it’s correctly defined by attempting to log in to thetarget node from the master:
$ ssh retailvm1d
You are authorizedto use this System for approved business purposes only.
Use for any otherpurpose is prohibited. All transactional records, reports,
email, software andother data generated by or residing upon this System,
to the extentpermitted by local law, are the property of Citigroup Inc.
or one of itssubsidiaries or their affiliates
(individually orcollectively ' Citigroup ') and may be used by Citigroup
for any purposeauthorized and permissible in your country of work.
Activities on thisSystem are monitored to the extent permitted by local law.
PAM Authentication
Password:
Last login: Mon Jul22 2013 07:15:29 -0400 from APACCNSHZJW2594.apac.nsroot.net
Change JAVA_HOME inhadoop-env.sh.
The standalone modeis the default mode for Hadoop. When you first uncompress the Hadoop source package,it’s ignorant of your hardware setup. Hadoop chooses to be conservative andassumes a minimal configuration.
分享到:
相关推荐
Hadoop Cluster Deployment.
Setting up Hadoop for Elasticsearch Setting up Java Setting up a dedicated user Installing SSH and setting up the certificate Downloading Hadoop Setting up environment variables Configuring ...
hadoop cluster build detail
Easy-to-understand recipes for securing and monitoring a Hadoop cluster, and design considerations Recipes showing you how to tune the performance of a Hadoop cluster Learn how to build a Hadoop ...
Hadoop cluster planning guide
ssh for hadoop
配置hadoop的集群文档,包含了详细配置的PDF文档和WordCount代码
hadoop-cluster-docker, 在 Docker 容器中运行 Hadoop 在 Docker 容器内运行Hadoop集群博客:在 Docker 更新中运行Hadoop集群。博客:基于Docker搭建Hadoop集群之升级版 3节点Hadoop集群 1.拉 Docker 图像sudo do
指导Hadoop集群部署的资料, 注意: 内容是英文的, 可能有些同学会失望
Spring Data for Apache Hadoop API。 Spring Data for Apache Hadoop 开发文档
NULL 博文链接:https://mryangjw.iteye.com/blog/2062690
Data Algorithms Recipes for Scaling Up with Hadoop and Spark 英文epub 本资源转载自网络,如有侵权,请联系上传者或csdn删除 本资源转载自网络,如有侵权,请联系上传者或csdn删除
SQL for Apache Hadoop, SQL for Apache Hadoop, SQL for Apache Hadoop, SQL for Apache Hadoop
Hadoop在centOS系统下的安装文档,系统是虚拟机上做出来的,一个namenode,两个datanode,详细讲解了安装过程。
基于Java和ssh在Hadoop平台上完成文件操作,结果查询等功能
ch09 - Setting Up a Hadoop Cluster ch10 - Administering Hadoop ch11 - Pig ch12 - Hive ch13 - HBase ch14 - ZooKeeper ch15 - Sqoop ch16 - Case Studies app1 - Installing Apache Hadoop app2 - Cloudera's ...
Chapters 9 and 10 are for Hadoop administrators, and describe how to set up and maintain a Hadoop cluster running HDFS and MapReduce. Chapters 11, 12, and 13 present Pig, HBase, and ZooKeeper, ...
MapReduce is the distribution system that the Hadoop MapReduce engine uses to distribute work around a cluster by working parallel on smaller data sets. It is useful in a wide range of applications, ...
Several good tools and guides describe how to deploy Hadoop clusters, but very little documentation tells how to increase performance on a Hadoop cluster once it is deployed. This white paper ...
Setting Up a Hadoop Cluster Chapter 11. Administering Hadoop Part IV. Related Projects Chapter 12. Avro Chapter 13. Parquet Chapter 14. Flume Chapter 15. Sqoop Chapter 16. Pig Chapter 17. Hive ...