Toolset for logging and monitoring

In this section, I will review some aspects on the log and monitoring related softwares such as syslog, nagios and ganglia.

rsyslog

logrotate

Logstash

Merged into elk stack

Pipeline from log to elastic search, see this blog for introduction.

Nagios

Installation

• See this post for its installation on ubuntu (one node).

• Install on ubuntu 18.04 cluster and basic configuration, post

Ganglia

Installation

• ubuntu16.04

Backend

rrdtool: round robin database

ELK

Elasticsearch + logstash+kibana, to be dployed soon, firstly install on VM environment

initial install finished on the cluster

simple intro

Installation of ELK on ubuntu18.04 by digital ocean

ganglia input plugin for logstash: doc

multiple input for logstash and type label: so

ganglia {
    port => 28649
    host => "{{ master_ip }}"
    type => "ganglia"

  }

  udp_send_channel {
  host = {{ master_name }}
  port = 28649
  ttl = 1
}

experiment conclusion: ganglia doesn't work well with logstash

elasticalert: combine ELK stack and email alert

Cluster of ES post, to be deployed. One also need to configure ssl for es cluster if authetication is enabled. see this doc.

Benchmark

Linpack (hpl)

post on linpack

Theorectial Flop value for CPU: post

All of the Gold 6000 processors have two AVX512 units, so they are capable of 32 DP FLOPS/cycle. The Gold 5000 processors have one AVX512 unit (except for the Gold 5122, which has two), so they are capable of 16 DP FLOPS/cycle.

Simply linpack directly from intel parallel studio: blog

Results: 580Gflop on single machine in normal env without fine tuning

Intel mkl linpack documentation: intel

MPI benchmarks

• An example using spack installed stack for mpi bandwidth benchmark: blog

IO benchmarks

• An example using spack installed stack for io benchmark: blog