Installation

You should have python 2.7.5 or 3.6 and pip installed.

Once obtained the project, create a virtual env.

Dependencies

Phantomas

Phantomas is a PhantomJS-based modular web performance metrics collector.

Phantomas official documentation

Requirements:

$ sudo npm install --global --no-optional phantomas phantomjs-prebuilt@^2.1.5

MongoDb

To install MongoDb follow your distro-specific instructions.

InfluxDB

From the InfluxDB official Documentation:

  • Ubuntu & Debian
$ wget https://dl.influxdata.com/influxdb/releases/influxdb_1.2.2_amd64.deb
$ sudo dpkg -i influxdb_1.2.2_amd64.deb
  • RedHat & CentOS
$ wget https://dl.influxdata.com/influxdb/releases/influxdb-1.2.2.x86_64.rpm
$ sudo yum localinstall influxdb-1.2.2.x86_64.rpm

Grafana*

*Not needed, useful for extracting graphical data from InfluxDB.

At https://grafana.com/grafana/download we found the following distro-specific commands:

  • Ubuntu & Debian
$ cd ~/hostit-monitoring/
$ wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.2.0_amd64.deb
$ sudo dpkg -i grafana_4.2.0_amd64.deb
  • RedHat & CentOS
$ cd ~/hostit-monitoring/
$ wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.2.0-1.x86_64.rpm
$ sudo yum localinstall grafana-4.2.0-1.x86_64.rpm

Project Installation

  • Install hostit agent:
$ pip install -e hostit
  • Run setup.sh script in the hostit-monitoring directory:
$ ./setup.sh

Database setup

MongoDb

Launch the MongoDb service:

$ sudo service mongodb start

In setup-model.py you can find an initial setup for MongoDb, including SLA, time, tolerance, websites and extra infos. setup-model.py can be modified as needed.

$ python hostit-monitoring/setup-model.py

Run setup-model.py only once, during the installation process.

InfluxDb

Change the configuration file (usually at /usr/local/etc/influxdb.conf or /etc/influxdb/influxdb.conf) to enable the UDP protocol on a free port.

In this example the port 4444 was arbitrarily chosen:

[[udp]]
    enabled = true
    bind-address = ":4444" # the bind address
    database = "hostit_monitoring" # Name of the database that will be written to
    batch-size = 1000 # will flush if this many points get buffered
    batch-timeout = "1s" # will flush at least this often even if the batch-size is not reached
    batch-pending = 5 # number of batches that may be pending in memory
    read-buffer = 0 # UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max.

Launch the InfluxDB server pointing to the above configuration file:

$ influxd -config=/usr/local/etc/influxdb.conf

Launch the influx client and create a database with the same name of the database parameter in the above configuration:

$ influx
> CREATE DATABASE `hostit_monitoring`

Initial configurations

Supervisor

The Supervisor Configuration File file is located at hostit-monitoring/hsm_supervisor/hsm_supervisor/development.ini

You’ll have to check:

  • supervisor_host
    the host where the Supervisor application will run.
  • supervisor_port
    the port where the Supervisor application will run. host and port have to be free from other services, the default should work.
  • default_worker_config
    points to the *Worker* config file on your filesystem. It’s the config.ini inside the hostit-monitoring/hsm_worker directory.
  • hsm_worker_utility_class_path
    path of the utility class for managing workers. See Worker Utility

The defaults are:

[DEFAULT]
supervisor_host = 127.0.0.1
supervisor_port = 8080
supervisor_check_time = 30
reload_queue_time = 30

mongodb_address = 127.0.0.1
mongodb_port = 27017

ming_address = 127.0.0.1
ming_port = 27017
ming_db = hsm

default_worker_config = /Users/axant/work-new/hostit-monitoring/hsm_worker/hsm_worker/config.ini
hsm_worker_utility_class_path= hsm_supervisor.lib.default_worker_utility:DefaultWorkerUtility


delay_percentage = 10
advance_percentage= 5


[loggers]
keys=root

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=

Worker

The Worker config file is located at hostit-monitoring/hsm_worker/hsm_worker/config.ini

You’ll have to check:

  • ming
    Refers to MongoDb, it should point to where the MongoDb service is listening
  • influxdb
    It should point to where the influxdb service is listening
  • supervisor_url
    It should point to the URL:port you set up in the Supervisor config file
  • failure_measurment_api
    It should point to the URL:port you set up in the Supervisor config file
  • sleep_time
    It’s the Worker sleeping time when there are no sites to check
  • max_concurrency_tasks
    It’s the maximum number of tasks for each Worker

The defaults are:

[DEFAULT]
ming_address = 127.0.0.1
ming_port = 27017
ming_db = hsm

influxdb_address = 127.0.0.1
influxdb_udp_port = 4444

sleep_time = 2
max_concurrency_tasks = 1
supervisor_url = http://127.0.0.1:8080/website/get_site

failure_measurment_api = http://127.0.0.1:8080/failure_measurment

[loggers]
keys=root

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=INFO
handlers=consoleHandler

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=

Installation is now complete, you are now ready to Start monitoring.