Prometheus
Prometheus, at its core, is a very simple and powerful tool to monitor and track your servers. Let’s take a use case. A company deploys and services 100s of ec2 instances on Amazon. How will that company be able to keep track of all the events and services that are running on those instances? What if one of those instances seem fine and is up and running, however, the core service it is meant for has crashed. The only way to check this is through the log files, but who will be responsible for checking through the log files of hundreds of instances?
This is where a service like Prometheus is so useful. Prometheus can monitor hundreds of instances and display metrics and statistics from all of them to you in one simple dashboard. You can even get more in-depth dashboards by connecting Grafana to your Prometheus service.
Grafana is a web application that gives you visualization tools. In a nutshell, you can create and use dashboards from Grafana to visualize all the statistics and metrics presented by Prometheus.
Project Scope
Today we will be creating an Ansible playbook to automate the creation of an EC2 instance as well as the download of grafana, node_exporter, and Prometheus into that instance. We will tackle this project in small steps.
Step 1
Manually create an ec2 box and download Prometheus, node_exporter, and grafana into and ensure everything is running.
Step 2
Create an Ansible playbook that will go through that same process in step 1.
Step 3
Write into the playbook the ability to create an ec2 instance.
End
With the run of a single playbook, an ec2 instance should be deployed with Prometheus, node_exporter, and grafana installed into it.
Step 1: Prometheus and Grafana Installation
Pre-requisites:
Download AWS CLI and run aws configure
Create an ec2 instance with Ubuntu 16.04
Make sure ports 80, 22, 9090, 9100, and 3000 are open on your ec2 instance
Ssh into your ec2 instance
Download nginx:
We will be using Nginx to host our web service. Follow these steps to download and install Nginx onto your Ubuntu box.
$ sudo apt-get update
$ sudo apt-get install nginx
After installing check your web server.
$ systemctl status nginx
The server should be active and running. Confirm this by going to your ec2 instance’s public ip. You should see a ‘Welcome to nginx!’ message.
Download prometheus:
Create two new users to isolate the ownership of Prometheus’ core files and directories.
$ sudo useradd --no-create-home --shell /bin/false prometheus
$ sudo useradd --no-create-home --shell /bin/false node_exporter
Create the directories to store Prometheus’ files and data.
$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus
Set the user and group ownership on the new directories to the prometheus user.
$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus
Download the latest version of prometheus: https://prometheus.io/download/
$ cd ~
$ curl -LO https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gz
Use the sha256sum command to generate a checksum and compare it to the checksum on the prometheus website to ensure correct download.
$ sha256sum prometheus-2.21.0.linux-amd64.tar.gz
Tar the download.
$ tar xvf prometheus-2.21.0.linux-amd64.tar.gz
Copy the binaries to the proper directories.
$ sudo cp prometheus-2.21.0.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.21.0.linux-amd64/promtool /usr/local/bin/
Set the user and group ownership on the binaries to the prometheus user created before.
$ sudo chown prometheus:prometheus /usr/local/bin/prometheus
$ sudo chown prometheus:prometheus /usr/local/bin/promtool
Copy the consoles and console_libraries directories to /etc/prometheus.
$ sudo cp -r prometheus-2.21.0.linux-amd64/consoles /etc/prometheus
$ sudo cp -r prometheus-2.21.0.linux-amd64/console_libraries /etc/prometheus
Set the user and group ownership on the directories to the prometheus user.
$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
Remove files that are no longer needed
$ rm -rf prometheus-2.21.0.linux-amd64.tar.gz prometheus-2.21.0.linux-amd64
Create a config file to run prometheus.
$ sudo nano /etc/prometheus/prometheus.yml
Copy the contents to the config file.
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
Set the user and group ownership on the configuration file to the prometheus user.
$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
Start up Prometheus as the prometheus user, providing the path to both the configuration file and the data directory.
$ sudo -u prometheus /usr/local/bin/prometheus \
$ --config.file /etc/prometheus/prometheus.yml \
$ --storage.tsdb.path /var/lib/prometheus/ \
$ --web.console.templates=/etc/prometheus/consoles \
$ --web.console.libraries=/etc/prometheus/console_libraries
Create a new service file.
$ sudo nano /etc/systemd/system/prometheus.service
Copy contents to service file.
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
Reload systemd.
$ sudo systemctl daemon-reload
Start Prometheus.
$ sudo systemctl start prometheus
Check the service status.
$ sudo systemctl status prometheus
Enable service to start on boot.
$ sudo systemctl enable prometheus
Download the latest version of node_exporter: https://prometheus.io/download/
$ cd ~
$ curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
Use the sha256sum command to generate a checksum and compare it to the checksum on the prometheus website to ensure correct download.
$ sha256sum node_exporter-1.0.1.linux-amd64.tar.gz
tar the download.
$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz
Copy files to correct directory and change ownership to node_exporter user created earlier.
$ sudo cp node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
Remove files which are no longer needed.
$ rm -rf node_exporter-1.0.1.linux-amd64.tar.gz node_exporter-1.0.1.linux-amd64
Create a service file for node_exporter.
$ sudo nano /etc/systemd/system/node_exporter.service
Copy the content into the service file.
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
Reload systemd.
$ sudo systemctl daemon-reload
Start node_exporter.
$ sudo systemctl start node_exporter
Check the status.
$ sudo systemctl status node_exporter
Enable node_exporter to start on boot.
$ sudo systemctl enable node_exporter
Open the config file once more and add to it.
$ sudo nano /etc/prometheus/prometheus.yml
Your config file should look like this.
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'node_exporter'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9100']
Restart prometheus.
$ sudo systemctl restart prometheus
Check the status.
$ sudo systemctl status prometheus
You should be able to enter the public IP now and you should see the prometheus dashboard.
Download the latest version of Grafana: https://grafana.com/grafana/download
$ sudo apt-get install -y adduser libfontconfig
$ wget https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb
$ sudo dpkg -i grafana_7.2.0_amd64.deb
Set up and start the Grafana server.
$ sudo systemctl daemon-reload && sudo systemctl enable grafana-server && sudo systemctl start grafana-server
Enter the public IP with port 3000 online to check the grafana dashboard. Add a new data source and select prometheus as the data source. Set the Prometheus server URL as the public IP URL with port 9090. Click add to test the connection and save the new data source.
You have successfully set up Prometheus and grafana on your ec2 instance manually. Our next step is to turn these command prompts into an Ansible playbook.
Part 2: Ansible
Ansible is a simple automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many others. It uses the YAML format to develop code for it. We will be creating many Ansible playbooks and roles to automate the installation process in part 1.
Directory Structure
roles
/grafana
/tasks
main.yml
/nginx
/tasks
main.yml
/node_exporter
/files
node_exporter.service
prometheus2.yml
/tasks
main.yml
/prometheus
/files
prometheus.service
prometheus.yml
/tasks
main.yml
/ubuntu_update
/tasks
main.yml
main.yml
Lets compare the command line prompts to the Ansible yaml files.
roles/ubuntu_update/tasks/main.yml
Command line:
$ sudo apt-get update
main.yml:
---
- name: Run the equivalent of "apt-get update" as a separate step
apt:
update_cache: yes
roles/nginx/tasks/main.yml
Command line:
$ sudo apt-get update
main.yml:
---
- name: Download nginx
apt:
pkg:
- nginx
roles/prometheus/tasks/main.yml
Command line:
$ sudo useradd --no-create-home --shell /bin/false prometheus
$ sudo useradd --no-create-home --shell /bin/false node_exporter
$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus
$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus
$ cd ~
$ curl -LO https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gz
$ sha256sum prometheus-2.21.0.linux-amd64.tar.gz
$ tar xvf prometheus-2.21.0.linux-amd64.tar.gz
$ sudo cp prometheus-2.21.0.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.21.0.linux-amd64/promtool /usr/local/bin/
$ sudo chown prometheus:prometheus /usr/local/bin/prometheus
$ sudo chown prometheus:prometheus /usr/local/bin/promtool
$ sudo cp -r prometheus-2.21.0.linux-amd64/consoles /etc/prometheus
$ sudo cp -r prometheus-2.21.0.linux-amd64/console_libraries /etc/prometheus
$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
$ rm -rf prometheus-2.21.0.linux-amd64.tar.gz prometheus-2.21.0.linux-amd64
$ sudo nano /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
$ sudo -u prometheus /usr/local/bin/prometheus \
$ --config.file /etc/prometheus/prometheus.yml \
$ --storage.tsdb.path /var/lib/prometheus/ \
$ --web.console.templates=/etc/prometheus/consoles \
$ --web.console.libraries=/etc/prometheus/console_libraries
$ sudo nano /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
$ sudo systemctl daemon-reload
$ sudo systemctl start prometheus
$ sudo systemctl status prometheus
$ sudo systemctl enable prometheus
main.yml:
---
- name: Create prometheus user
user:
name: prometheus
state: present
shell: /bin/false
system: no
createhome: no
become_user: root
- name: Create directory
file:
path: "{{ item }}"
state: directory
owner: prometheus
group: prometheus
with_items:
- /etc/prometheus
- /var/lib/prometheus
- name: Unarchive a file that needs to be downloaded
unarchive:
src: https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gz
dest: /home/ubuntu
remote_src: yes
- name: Copy file
copy:
src: /home/ubuntu/prometheus-2.21.0.linux-amd64/prometheus
dest: /usr/local/bin/
- name: Copy file
copy:
src: /home/ubuntu/prometheus-2.21.0.linux-amd64/promtool
dest: /usr/local/bin/
- name: Change file permission
file:
path: /usr/local/bin/prometheus
owner: prometheus
group: prometheus
mode: '755'
- name: Change file permission
file:
path: /usr/local/bin/promtool
owner: prometheus
group: prometheus
- name: Copy file
copy:
src: /home/ubuntu/prometheus-2.21.0.linux-amd64/consoles
dest: /etc/prometheus
- name: Copy file
copy:
src: /home/ubuntu/prometheus-2.21.0.linux-amd64/console_libraries
dest: /etc/prometheus
- name: Change file permission
file:
path: /etc/prometheus/consoles
owner: prometheus
group: prometheus
- name: Change file permission
file:
path: /etc/prometheus/console_libraries
owner: prometheus
group: prometheus
- name: Delete unwanted files
file:
state: absent
path: /home/ubuntu/prometheus-2.21.0.linux-amd64
- name: Delete unwanted files
file:
state: absent
path: /home/ubuntu/prometheus-2.21.0.linux-amd64.tar.gz
- name: Copy file
copy:
src: /home/ubuntu/ansible-grafana/roles/prometheus/files/prometheus.yml
dest: /etc/prometheus/prometheus.yml
- name: Change file permission
file:
path: /etc/prometheus/prometheus.yml
owner: prometheus
group: prometheus
- name: Copy file
copy:
src: /home/ubuntu/ansible-grafana/roles/prometheus/files/prometheus.service
dest: /etc/systemd/system/prometheus.service
- name: Start prometheus
systemd:
state: started
name: prometheus
daemon_reload: yes
enabled: yes
We also have a couple of service and setup files in the prometheus directory.
roles/prometheus/files/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
roles/prometheus/files/prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
roles/node_eporter/tasks/main.yml
Command line:
$ cd ~
$ curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
$ sha256sum node_exporter-1.0.1.linux-amd64.tar.gz
$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz
$ sudo cp node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
$ rm -rf node_exporter-1.0.1.linux-amd64.tar.gz node_exporter-1.0.1.linux-amd64
$ sudo nano /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
$ sudo systemctl daemon-reload
$ sudo systemctl start node_exporter
$ sudo systemctl status node_exporter
$ sudo systemctl enable node_exporter
$ sudo nano /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'node_exporter'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9100']
$ sudo systemctl restart prometheus
$ sudo systemctl status prometheus
main.yml:
---
- name: Create node_exporter user
user:
name: node_exporter
state: present
shell: /bin/false
system: no
createhome: no
become_user: root
- name: Unarchive a file that needs to be downloaded
unarchive:
src: https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
dest: /home/ubuntu
remote_src: yes
- name: Copy file
copy:
src: /home/ubuntu/node_exporter-1.0.1.linux-amd64/node_exporter
dest: /usr/local/bin/
- name: Change file permission
file:
path: /usr/local/bin/node_exporter
owner: node_exporter
group: node_exporter
mode: '755'
- name: Delete unwanted files
file:
state: absent
path: /home/ubuntu/node_exporter-1.0.1.linux-amd64.tar.gz
- name: Delete unwanted files
file:
state: absent
path: /home/ubuntu/node_exporter-1.0.1.linux-amd64
- name: Copy file
copy:
src: /home/ubuntu/ansible-grafana/roles/node_exporter/files/node_exporter.service
dest: /etc/systemd/system/node_exporter.service
- name: Start node_exporter
systemd:
state: started
name: node_exporter
daemon_reload: yes
enabled: yes
- name: Copy file
copy:
src: /home/ubuntu/ansible-grafana/roles/node_exporter/files/prometheus2.yml
dest: /etc/prometheus/prometheus.yml
- name: Restart prometheus
systemd:
name: prometheus
state: restarted
We also have a couple of service and setup files in the prometheus directory.
roles/node_exporter/files/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
roles/node_exporter/files/prometheus2.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'node_exporter'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9100']
roles/grafana/tasks/main.yml
Command line:
$ sudo apt-get install -y adduser libfontconfig
$ wget https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb
$ sudo dpkg -i grafana_7.2.0_amd64.deb
$ sudo systemctl daemon-reload && sudo systemctl enable grafana-server && sudo systemctl start grafana-server
main.yml:
---
- name: Download adduser libfontconfig
apt:
pkg:
- adduser
- libfontconfig
- name: Download grafana
apt:
deb: https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb
- name: Start grafana
systemd:
state: started
name: grafana-server
daemon_reload: yes
enabled: yes
./main.yml
- name: run roles on remote box
hosts: localhost
become: true
roles:
- ubuntu_update
- github
- nginx
- prometheus
- node_exporter
- grafana
Part 3: AWS automation
The next step is to automate the creation of the aws ec2 instance as well as push the code to github and pull it into the remote ec2 box. Therefore the directory structure is updated as well as the ./main.yml file.
roles
/github
/tasks
main.yml
/grafana
/tasks
main.yml
/nginx
/tasks
main.yml
/node_exporter
/files
node_exporter.service
prometheus2.yml
/tasks
main.yml
/prometheus
/files
prometheus.service
prometheus.yml
/tasks
main.yml
/ubuntu_update
/tasks
main.yml
main.yml
./main.yml
---
- name: Create an EC2 instance
hosts: localhost
gather_facts: False
tasks:
- name: Launch instance
ec2:
key_name: #enter your key name
group: #enter your security group
instance_type: #enter instance type
image: #enter image id
wait: true
region: #enter region
register: ec2
- name: Print all ec2 variables
debug: var=ec2
- name: Get theh Ip address
debug: var=ec2.instances[0].public_dns_name
- name: add host to group 'just_created' with variable foo=42
add_host:
name: "{{ ec2.instances[0].public_dns_name }}"
groups: ec2_hosts
ansible_host: "{{ ec2.instances[0].public_dns_name }}"
ansible_ssh_user: ubuntu
ansible_ssh_private_key_file: ~/#enter key location
- name: Wait for a while
pause: seconds=60
- name: run roles on remote box
hosts: ec2_hosts
become: true
roles:
- ubuntu_update
- github
- nginx
- prometheus
- node_exporter
- grafana
roles/github/tasks/main.yml
---
- name: Clone a repo
git:
repo: https://github.com/ansible/ansible-examples.git
dest: /home/ubuntu
Run the command:
sudo ansible-playbook main.yml
This should go ahead and create an ec2 box as well as download Prometheus and Grafana into that box.
Congratulations on automating the Prometheus deployment with Ansible!
Comments