Sun Apr 22, 2018 6:17 pm
Login Register Lost Password? Contact Us


Installing Ganglia Ubuntu 14.04

Questions related to node architecture, redundancy and system monitoring

Tue Oct 18, 2016 11:51 am Change Time Zone

Hi Gleb,

Apologies with the late response to your post, I unfortunately had to work on some other projects over the last few weeks.

I haven’t had a chance to look into the remote mount approach yet, but will be getting back into this over the next few days.

However, I have had a strange problem occur on one of our live Roxies that I have Ganglia installed on.

All of a sudden last week, the Ganglia service / Gmond was using up 100% CPU which caused the Roxie to become unresponsive.

I have rebooted it, but the problem occurs pretty much straight away and nothing has been changed since we last spoke in August.

I can get around the problem by running sudo service ganglia-monitor stop but would like to know if you have come across this before?

I have been hunting around for log files that may shed some light on this but have found nothing at the moment,

Do you know of any log files that may contain Gmond / Ganglia information?

I have attached a screen shot of process list for you as well.

Best Regards

Antony
Attachments
Screen Shot 2016-10-18 at 12.37.00.png
Screen Shot 2016-10-18 at 12.37.00.png (43.31 KiB) Viewed 858 times
amillar
 
Posts: 14
Joined: Fri Oct 16, 2015 7:32 am

Mon Dec 18, 2017 3:14 pm Change Time Zone

Hi,

We've now got Ganglia Monitoring on all of our production our single-node Roxies.

Now that we have this, we would like to roll ganglia out to our Multi-Node Roxies.

What are the best practices for monitoring multi-nodes through Ganglia?

Do we only need to install Ganglia on the node which clients hit (Master node of the Multi-Node cluster) or do we need to install this on all nodes within the multi-node cluster?

Thanks,
Antony
amillar
 
Posts: 14
Joined: Fri Oct 16, 2015 7:32 am

Mon Dec 18, 2017 4:27 pm Change Time Zone

Hi Anthony,

We generally suggest to follow Ganglia's documentation for install.

That being said, Gmond is generally installed across all nodes, and Gmetad is installed on your master node.

Once, you have that configured per Ganglia's or your own systems requirements, the Ganglia-Monitoring Plugin provided by HPCC should work to display data in ECLWatch.

Thanks,
Kunal
KunalA
 
Posts: 3
Joined: Tue Feb 18, 2014 8:15 pm

Tue Dec 19, 2017 9:56 am Change Time Zone

Hi Kunal,

Thanks for your very quick response, it's really appreciated.

Here's a little explanation of our current setup.

We currently have 4 Roxie clusters all being used by different external applications/clients.

On the Ganglia host we are running 4 different Gmond instances to act as the "Cluster" master for each of our Roxie Clusters. We are doing this by running gmond in Docker all running on different ports to separate them out between the different clusters.

We do this to keep all Ganglia data central to one host as we sometimes reload/upgrade hardware on our Roxie nodes.

These 4 different Ganglia instances running in Docker that then talk to the main Ganglia Master which runs another instance of Gmond and gmetad.

My Thoughts/Plan to enable the MultiNodes to work are as follows...
1. Install Ganglia gmond and HPCC Ganglia plugin on all nodes within the MultiNode.
2. Create a new Cluster for the MultiNode on the Ganglia Master host
3. Ensure that the Cluster gmond speaks to the Grid/Gmetad on the Ganglia Master host
4. Ensure that all nodes within the MultiNode speak to the newly created Cluster

So far, the same steps and process that has been followed for all previous instances.

As the gmond can send data to more than one host we would setup a gmetad instance on the Master node for the MultiNode. Once this is setup/configured we could then use ECL Watch on the Master node for the MultiNode to view all requests that the MultiNode is performing.

On another note:
As we are using a single host for all of our Ganglia data, we do not have the data rendering in ECL Watch. We have the plugin installed on each of our nodes but do not use the frontend provided by ECL Watch. Are there any benefits/additional capabilities of using the ECL Watch frontend over the Ganglia Frontend interface we are currently using?

Thanks,
Antony
amillar
 
Posts: 14
Joined: Fri Oct 16, 2015 7:32 am

Tue Dec 19, 2017 6:30 pm Change Time Zone

Hi Anthony,

The GangliaMonitoring-Plugin to ECLWatch is provided as a convenience, so that you have access to ganglia information without having to leave ECLWatch. It also enables a (red/yellow/green) indication light in the banner of ECLWatch. However, the Ganglia interface, may provide more features.

Thanks,
Kunal
KunalA
 
Posts: 3
Joined: Tue Feb 18, 2014 8:15 pm

Previous

Return to System Health

Who is online

Users browsing this forum: No registered users and 1 guest

cron