Tag Archives: gmond

Adding Python support for Ganglia – gmond on Centos / RHEL

While creating Ganglia RPM packages for my Centos repository I got stuck with a lack of Python support in Ganglia – gmond. I’ll try to to explain details of this whole issue.

First of all one could just install ganglia-gmond packages from Epel repo – and that will make the general case. I checked EPEL packages and those contain modpython.so so go for it ;) But I just couldn’t go that way as I had to create my own packages.

First of all – let’s check the compilation params regarding to Python:

[root@vm-2-repo ganglia-gmond-3.6.0]# ./configure --help | grep -i python
  --disable-python        exclude mod_python and support for metric modules written in python
  --with-python=PATH      Specify prefix for python or full path to interpreter

[root@vm-2-repo ganglia-gmond-3.6.0]# ./configure --help | grep -i python

--disable-python exclude mod_python and support for metric modules written in python

--with-python=PATH Specify prefix for python or full path to interpreter

So by default there should be Python support builtin. But after compiling & building there’s no modpython.so library on /usr/lib64/ganglia and that is weird. Digging deeper in config.log there’s:

configure:11482: checking for python
configure:11500: found /usr/bin/python
configure:11512: result: /usr/bin/python
configure:11528: checking Python version
configure:11532: result: 2.6
configure:11554: checking Python support
configure:11556: result: no

configure:11482: checking for python

configure:11500: found /usr/bin/python

configure:11512: result: /usr/bin/python

configure:11528: checking Python version

configure:11532: result: 2.6

configure:11554: checking Python support

configure:11556: result: no

Hmm – that’s really weird. Let’s go even more deeper – now with configure.ac:

AC_ARG_WITH( python,
[  --with-python=PATH      Specify prefix for python or full path to interpreter],
[if test x"$withval" != xno; then enable_python="yes"; PYTHON_BIN="$withval"; fi])

[...]

AC_ARG_ENABLE( python,
[  --disable-python        exclude mod_python and support for metric modules written
 in python],
[ if test x"$enableval" != xyes; then enable_python="no"; fi ], [ enable_python="yes" ] )

AC_ARG_WITH( python,

[ --with-python=PATH Specify prefix for python or full path to interpreter],

[if test x"$withval" != xno; then enable_python="yes"; PYTHON_BIN="$withval"; fi])

[...]

AC_ARG_ENABLE( python,

[ --disable-python exclude mod_python and support for metric modules written

in python],

[ if test x"$enableval" != xyes; then enable_python="no"; fi ], [ enable_python="yes" ] )

Ok so it’s true – by default there should be Python support. Let’s check further (still configure.ac):

if test x"$enable_python" = xyes; then
  echo
  echo Checking for python

  # check for Python executable
  if test -z "$PYTHON_BIN"; then
    AC_PATH_PROG(PYTHON_BIN, python)
  else
    if test -d "$PYTHON_BIN"; then
      PYTHON_BIN="$PYTHON_BIN/bin/python"
    fi
  fi

  if test -n "$PYTHON_BIN"; then
    # find out python version
    AC_MSG_CHECKING(Python version)
    PyVERSION=`$PYTHON_BIN -c ['import sys; print sys.version[:3]'`]
    PyMAJVERSION=`$PYTHON_BIN -c ['import sys; print sys.version[:1]'`]
    AC_MSG_RESULT($PyVERSION)
    PYTHON_VERSION=$PyVERSION
    AC_SUBST(PYTHON_VERSION)

    PyEXEC_INSTALLDIR=`$PYTHON_BIN -c "import sys; print sys.exec_prefix"`
    if test -f "$PyEXEC_INSTALLDIR/include/python/Python.h"; then
      PYTHON_INCLUDES="-I$PyEXEC_INSTALLDIR/include/python"
    else
      if test -f "$PyEXEC_INSTALLDIR/include/python$PyVERSION/Python.h"; then
        PYTHON_INCLUDES="-I$PyEXEC_INSTALLDIR/include/python$PyVERSION"
      else
        PYTHON_INCLUDES=""
        enable_python="no"
      fi
    fi
    AC_SUBST(PYTHON_INCLUDES)
  else
    enable_python="no"
  fi
fi

if test x"$enable_python" = xyes; then

echo

echo Checking for python

# check for Python executable

if test -z "$PYTHON_BIN"; then

AC_PATH_PROG(PYTHON_BIN, python)

else

if test -d "$PYTHON_BIN"; then

PYTHON_BIN="$PYTHON_BIN/bin/python"

if test -n "$PYTHON_BIN"; then

# find out python version

AC_MSG_CHECKING(Python version)

PyVERSION=`$PYTHON_BIN -c ['import sys; print sys.version[:3]'`]

PyMAJVERSION=`$PYTHON_BIN -c ['import sys; print sys.version[:1]'`]

AC_MSG_RESULT($PyVERSION)

PYTHON_VERSION=$PyVERSION

AC_SUBST(PYTHON_VERSION)

PyEXEC_INSTALLDIR=`$PYTHON_BIN -c "import sys; print sys.exec_prefix"`

if test -f "$PyEXEC_INSTALLDIR/include/python/Python.h"; then

PYTHON_INCLUDES="-I$PyEXEC_INSTALLDIR/include/python"

else

if test -f "$PyEXEC_INSTALLDIR/include/python$PyVERSION/Python.h"; then

PYTHON_INCLUDES="-I$PyEXEC_INSTALLDIR/include/python$PyVERSION"

else

PYTHON_INCLUDES=""

enable_python="no"

AC_SUBST(PYTHON_INCLUDES)

else

enable_python="no"

Ah crap – and here it is. Python support will be disabled and modpython.so library won’t be created when there’s no header files for Python on our OS. You can make sure if there is Python.h by using locate / find / whatever.

Solution? Just:

yum install python-devel

1	yum install python-devel

And that will make it.

SELinux && Ganglia / Multicast && Apache && RRDs

Posted by docent on September 29, 2013 4 comments

While setting up test server with Ganglia triad (gmetad, gmond, gweb over Apache) I had only 2 SELinux alerts to solve. First one regarding to denial of /var/lib/ganglia/rrd access by httpd process (while accessing ganglia-web interface):

type=AVC msg=audit(1380480812.081:1792): avc:  denied  { setattr } for  pid=5876 comm="rrdtool" name="fontconfig" dev=dm-1 ino=47054911 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:fonts_cache_t:s0 tclass=dir

1	type=AVC msg=audit(1380480812.081:1792): avc: denied { setattr } for pid=5876 comm="rrdtool" name="fontconfig" dev=dm-1 ino=47054911 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:fonts_cache_t:s0 tclass=dir

In order to resolve that I just added SELinux contexts to rrdtool binary and /var/lib/ganglia/rrds dir:

chcon -t httpd_sys_script_exec_t /usr/bin/rrdtool
chcon -t httpd_sys_content_t /var/lib/ganglia/rrds -R

1 2	chcon -t httpd_sys_script_exec_t /usr/bin/rrdtool chcon -t httpd_sys_content_t /var/lib/ganglia/rrds -R

The last SELinux alert was regarding to Apache trying to access port 8652 by socket connection (on ganglia-web I saw error like “fsockopen 8652 access denied”):

type=AVC msg=audit(1380480626.847:1764): avc:  denied  { name_connect } for  pid=2016 comm="httpd" dest=8652 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

1	type=AVC msg=audit(1380480626.847:1764): avc: denied { name_connect } for pid=2016 comm="httpd" dest=8652 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

So I had to create local policy for this particular occasion as there is no ganglia modules for SELinux (maybe I should create one…?). Firstly let’s check that there’s no even one module:

[root@netrunner audit]# semodule -l | egrep -i "(ganglia|gmond|gmetad|gweb)" | wc -l
0

1 2	[root@netrunner audit]# semodule -l \| egrep -i "(ganglia\|gmond\|gmetad\|gweb)" \| wc -l 0

So let’s create this policy:

[root@netrunner audit]# grep httpd /var/log/audit/audit.log | audit2allow -M http-gangliagmetad-port
[root@netrunner audit]# semodule -i http-gangliagmetad-port.pp

1 2	[root@netrunner audit]# grep httpd /var/log/audit/audit.log \| audit2allow -M http-gangliagmetad-port [root@netrunner audit]# semodule -i http-gangliagmetad-port.pp

After this everything should work like a charm. To confirm that We have this new policy working:

[root@netrunner modules]# semodule -l | grep ganglia
http-gangliagmetad-port	1.0

1 2	[root@netrunner modules]# semodule -l \| grep ganglia http-gangliagmetad-port 1.0

Ganglia, multicast && KVM on CentOS

Posted by docent on August 9, 2013 No comments

This is just a short note – I have to post it as this problem was really annoying and I couldn’t find any solutions in Google, so had to resolve it by myself.

Don’t know what Ganglia is? Check here: http://ganglia.sourceforge.net/ – it just kicks ass :)

Problem? I installed gmond on all our hosts / guests (CentOS 5/6, KVM virt, latest Ganglia daemons), also configured properly gmetad daemons and started this whole stuff using multicast. And it was working – for a while. After about 10-20 minutes it just stopped working on KVM guests. I saw no charts for those machines – but gmonds (even in debug mode) didn’t reveal any problems. And KVM hosts’ charts were fine (mostly..).

One more thing – in KVM guests I always set “deaf = yes” (just don’t want to have too much multicast traffic – i set it to “no” only on some bare hosts).

Ok so the problem.. I hung for some time on tcpdump / strace and came to the root of this problem – somehow there was no multicast traffic on KVM guests (I turned off iptables on KVM guests for the time of this whole issue – resolving). After some time I found 2 possible root causes:

On KVM hosts by default there is multicast filter set on: no-ip-multicast (You can check if You have it turned on with following command:

virsh nwfilter-list | grep 'no-ip-multicast'

1

virsh nwfilter-list | grep 'no-ip-multicast'

If it’s turned on – You can turn it off with:

virsh nwfilter-undefine no-ip-multicast service libvirtd restart

1
2

virsh nwfilter-undefine no-ip-multicast
service libvirtd restart

And that should do this part of the trick
And also – on CentOS KVM guests we have to turn off rp_filter in /etc/sysctl.conf:

net.ipv4.conf.default.rp_filter = 0

1

net.ipv4.conf.default.rp_filter = 0

and:

sysctl -p

1

sysctl -p

(You can try setting it to “loose mode” (so value: 2) instead of 0 – it can work for You and it’s always safer

Thats all for now. My sources for this one?

Always in use: tcpdump / strace ;)
Oh my – KVM why do you do this for me (“for security You fool!”)? About KVM firewall: http://libvirt.org/firewall.html
More about security and rp_filter: https://access.redhat.com/site/solutions/53031
And this nice RFC: http://www.ietf.org/rfc/rfc3704.txt

SecOPS / SysOp blog

The desire for safety stands against every great and noble enterprise.

Tag Archives: gmond

Adding Python support for Ganglia – gmond on Centos / RHEL

SELinux && Ganglia / Multicast && Apache && RRDs

Ganglia, multicast && KVM on CentOS

Pages

Recent Posts

SecOPS / SysOp blog

The desire for safety stands against every great and noble enterprise.

Tag Archives: gmond

Adding Python support for Ganglia – gmond on Centos / RHEL

SELinux && Ganglia / Multicast && Apache && RRDs

Ganglia, multicast && KVM on CentOS

Pages

Recent Posts

Tags