使用Ganglia监控Spark 2015-12-11 02:57:49 由于软件协议问题，Spark build默认不包含Ganglia的编译，使用Ganglia监控Spark首先需要重新编译Spark，将GangliaSink编译到Spark中。 使用sbt编译时，加入 -Pspark-ganglia-lgpl ``` build/sbt -Pyarn -Phadoop-2.4 -Phive -Phive-thriftserver -Pspark-ganglia-lgpl assembly ``` ## 附：Ganglia的安装配置 安装epel ``` rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm ``` 修改epel配置, 修改文件“/etc/yum.repos.d/epel.repo”， 将baseurl的注释取消， mirrorlist注释掉 中心节点安装（yum install -y xxx）： ``` gmetad: ganglia-gmetad gmond: ganglia-gmond ganglia-web: ganglia-web Apache: httpd php: php ``` 被监测节点： ``` gmond：ganglia-gmond ``` 中心节点Ganglia-web配置 ``` ln -s /usr/share/ganglia /var/www/html chown -R apache:apache /var/www/html/ganglia chmod -R 755 /var/www/html/ganglia ``` 访问权限：修改/etc/httpd/conf.d/ganglia.conf ``` <Location /ganglia> Order deny,allow # Deny from all # Allow from 127.0.0.1 # Allow from ::1 Allow from all # Allow from .example.com </Location> ``` rrd数据库权限 ``` chown -R ganglia:ganglia /var/lib/ganglia/rrds ``` 配置/etc/ganglia/gmetad.conf ``` data_source “ClusterName” masterIP:8649 ``` 配置/etc/ganglia/gmond.conf ``` cluster { name = “ClusterName” owner = “ganglia" ... } udp_send_channel { # the host who gather this cluster's monitoring data and send these data to gmetad node host = masterIP port = 8649 } udp_recv_channel { port = 8649 } tcp_accept_channel { port = 8649 } ``` 中心节点启动 ``` service httpd start service gmetad start service gmond start ``` 自启动 ``` chkconfig httpd on chkconfig gmetad on chkconfig gmond on ``` 被检测节点 ``` service gmond start chkconfig gmond on ``` 非特殊说明，均为原创，原创文章，未经允许谢绝转载。 原始链接：使用Ganglia监控Spark 赏 Prev 4K显示器入手指南 Next Hive注册UDF的问题
