ADD_TEXTILE("getting_backtrace.textile" "Spectrum 2 - Getting a backtrace")

ADD_TEXTILE("munin.textile" "Spectrum 2 - Munin integration")

If Spectrum is crashing, it’s useful to get backtrace to help us to find the reason. To get a backtrace you have to have debugging symbols installed or compiled Spectrum with them.

h2. Installing debugging symbols

a) If you are installing from our Debian/Ubuntu repository, you can just install debugging symbols with this command:

<pre>

sudo apt-get install spectrum2-dbg libtransport-dbg

</pre>

*Note:* The debug package has to be in the exact same version as the main package. So your spectrum installation might be upgraded as well when you install these packages.

b) If you build Spectrum by yourself, you have to build it in Debug mode.

<pre>

cmake . -DCMAKE_BUILD_TYPE=Debug

make

sudo make install

</pre>

h2. Installing GDB

<pre>

sudo apt-get install gdb

</pre>

h2. Getting a backtrace from a coredump

This is preferred method how to get the backtrace, because Spectrum runs without performance issues and once it crashes, it generates a coredump. 

Reproduce the crash and Spectrum will generate the coredump (file named like “core.12345” where the number is Spectrum process ID) in the working_dir (that directory is configurable in config file, default value is /var/lib/spectrum2/$jid/). Now you just have to get the backtrace from the coredump:

<pre>

cd /var/lib/spectrum/$jid/userdir

gdb spectrum2 core.12345

bt full

</pre>

h2. Getting a backtrace by running Spectrum in GDB

This is harder method how to get backtrace and also running Spectrum in GDB brings performance issues. Run Spectrum in GDB:

<pre>

gdb --args spectrum2 -n config_name

</pre>

where “config_name” is name of config you have in /etc/spectrum (You can also specify full path to config instead of its name).

You will see something like this:

<pre>

GNU gdb (GDB) 7.0-ubuntu

Copyright (C) 2009 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "i486-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /home/hanzz/code/test/transport/spectrum...done.

(gdb)

</pre>

Now you have to start Spectrum with following GDB command:

<pre>

run

</pre>

Since now Spectrum is running and you have to reproduce the crash or just wait for crash. Then get a backtrace with this GDB command:

<pre>

bt full

</pre>

* "Getting a backtrace":getting_backtrace.html

Munin is tool for collecting various information from system and showing them in charts. Spectrum 2 contains munin plugin which can be used to generate useful charts in Munin.

h2. Configuration

There's Munin plugin installed in by default in @/usr/share/munin/plugins/spectrum2_@. You have to create symlinks pointing to that files in @/etc/munin/plugins@ name like this:

|_. Symlink name |_. Meaning |

| @spectrum2_uptime@ | Uptime |

| @spectrum2_backends_count@ | Backends count |

| @spectrum2_crashed_backends_count@ | Crashed backends count |

| @spectrum2_online@ | Online users count |

| @spectrum2_messages@ | Total messages send over spectrum since the last restart |

| @spectrum2_messages_sec@ | Messages send per second over spectrum transports |

| @spectrum2_memory@ | Memory usage of transports |

| @spectrum2_average_memory_per_user@ | Average memory usage of per user |