Kensuke Kousaka's Blog

Notes for Developing Software, Service.

Build log for notifying server error system using Zabbix + Slack

To check server availability, build monitoring system using Zabbix + Slack.

Zabbix server (Monitoring server)

To build Zabbix, need to struct LAMP environemt.

Struct LAMP environment

Run following command to install packages related to LAMP.

# pacman -S apache php php-apache mariadb

After installing, configure Apache.

Apache

Edit Apache configuration file placed at /etc/httpd/conf/httpd.conf. In this time, I need to use only HTTPS so comment out unneeded ServerName and Listen 80.

Run following commands to issue SSL certificate.

# openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out server.key
# chmod 600 server.key
# openssl req -new -key server.key -out server.csr
# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

After issuing, Edit httpd.conf to support SSL. Append following 3 lines.

LoadModule ssl_module modles/mod_ssl.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
Include conf/extra/httpd-ssl.conf

After editting, run following commands to start / auto start when system startup.

# systemctl start httpd
# systemctl enable httpd

PHP

Edit httpd.conf to work PHP on Apache. Modify following line

LoadModule mpm_event_module modules/mod_mpm_event.so

to

LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

And add following line to end of line at LoadModule.

LoadModule php5_module modules/libphp5.so

Finally add following line to end of line at Include.

Include conf/extra/php5_module.conf

MariaDB

Run following commands to start MariaDB.

# mysql_install_db --user=mysql --basedir=/usr --datadir=/var/lib/mysql
# systemctl start mysqld

After starting service, run following command to initialize database.

# mysql_secure_installation

After initialization, uncomment follow line at /etc/mysql/my.cnf to avoid login through network.

skip-networking

After uncomment, run following commands to restart / auto start at system startup mysqld.

# systemctl restart mysqld
# systemctl enable mysqld

Zabbix

Run following command to install Zabbix.

# pacman -S zabbix-server-mysql

Create Apache configuration file /etc/httpd/conf/extra/zabbix.conf to access to Zabbix through Apache.

NameVirtualHost *:443
<IfModule mod_alias.c>
   Alias /zabbix /usr/share/webapps/zabbix/
</IfModule>

<Directory /usr/share/webapps/zabbix>
   Options -Indexes +FollowSymlinks +ExecCGI
    AllowOverride all
    Require all granted
</Directory>

<VirtualHost *:443>
   ServerAdmin foo@foovar.com
    DocumentRoot "/usr/share/webapps/zabbix"
    ServerName zabbix.your.domain
    ErrorLog /var/log/httpd/zabbix.your.domain-error_log
    CustomLog /var/log/httpd/zabbix.your.domain-access_log common

    SSLEngine on
    SSLCertificateFile "/etc/httpd/conf/server.crt"
    SSLCertificateKeyFile "/etc/httpd/conf/server.key"
</VirtualHost>

After creating file, read this configuration by appending following line to end of line at httpd.conf Include section.

Include conf/extra/zabbix.conf

After appending, edit /etc/php/php.ini by following to configure PHP for Zabbix working.

max_execution_time = 300
max_input_time = 300
post_max_size = 16M
always_populate_raw_post_data = -1
extension = bcmath.so
extension = gd.so
extension = mysqli.so
extension = pdo_mysql.so
extension = sockets.so
date.timezone = "America/Los_Angeles"

After configuring PHP, login MySQL by running following command.

$ mysql -u root -p

After login, run following commands to create Zabbix DB user account.

create database zabbix;
create user zabbix identified by 'YOUR_PREFERRED_PASSWORD';
grant all on zabbix.* to zabbix@localhost identified by "YOUR_PREFERRED_PASSWORD";

After running these query, run exit; to logout from MySQL, and run following commands to import Zabbix DB templates.

$ mysql -u zabbix -p zabbix < /usr/share/zabbix/database/schema.sql
$ mysql -u zabbix -p zabbix < /usr/share/zabbix/database/images.sql
$ mysql -u zabbix -p zabbix < /usr/share/zabbix/database/data.sql

Maybe you need to input password each time of running commands. Maybe prompt you to input password each time of running commands. You need to input YOUR_PREFERRED_PASSWORD specified when configuring Zabbix DB user.

Next, edit /etc/zabbix/zabbix_server.conf by following to configure Zabbix server.

DBHost = localhost
DBName = zabbix
DBUser = zabbix
DBPassword=YOUR_PREFERED_PASSWORD

After editting, start zabbix-server and configuring auto start at system startup.

# systemctl start zabbix-server
# systemctl enable zabbix-server

If you use firewall system, you need to open port 10051 to communicate Zabbix Agent (monitored server).

Access Zabbix web frontend with URL which configured at /etc/httpd/conf/extra/zabbix.conf. When prompt you to input Username/Password, input default Zabbix user Admin/zabbix to login.

Zabbix Agent (monitored server)

Run following command to install zabbix-agent to server which you want to monitor.

# pacman -S zabbix-agent

After installing, edit /etc/zabbix/zabbix_agentd.conf by following.

Server = IP address of Zabbix Server
ServerActive = IP address of Zabbix Server
Hostname = This server's hostname

After configuration, open port 10050 to communicate Zabbix Server if you use firewall system. Run following commands to start zabbix-agentd and configuring auto start at system startup.

# systemctl start zabbix-agentd
# systemctl enable zabbix-agentd

Notify server alert through Slask

Configure to send notification to Slack when something occurred at monitored server.

First, configure Incoming Webhook for your desired slack team and write down Webhook URL. Second, create Zabbix alertscript for Slack like following, give permission to execute, and place it to /usr/share/zabbix/alertscripts/.

Zabbix alertscript for Slack

Finally, create action of Zabbix. Create Configuration -> Actions by following.

  • Event Sources: Triggers
  • Operations
    • Operation type: Remote command
    • Target list: Current host
    • Type: Custom script
    • Execute on: Zabbix server

And input following script to Commands.

/usr/share/zabbix/alertscripts/zabbixslack.sh "{TRIGGER.STATUS}" "{TRIGGER.NAME}" "{ITEM.NAME1} {ITEM.KEY1}"

Setup has been completed. When Zabbix detecting server error, notify through Slack with some info.