Prometheus Pulls/Pushes Metrics from/to Another Prometheus

calendar Aug 31, 2022 · 3 min read · Prometheus  ·
Share on: linkedin copy

Overview

Requirement

I have 3 Prometheus servers and want to consolidate all of them into 1 Grafana. And one VM (Server C) is in a restricted zone and this zone only allows a special firewall rule from this zone to other zones.

Prometheus A pulls (receives) metrics from Prometheus B

In this scenario, we use the Federation feature to pull metrics from another Prometheus. You don't need to make any changes to Prometheus B, just add a new job to prometheus.yml of Prometheus A.

Please check lines 13~15, you can only pull some metrics you need. 

 1# prometheus.yml of A
 2  - job_name: 'federate'
 3    scrape_interval: 15s
 4    honor_labels: true
 5    metric_relabel_configs:
 6      - source_labels: [id]
 7        regex: '^static-agent$'
 8        action: drop 
 9    metrics_path: '/federate'
10    params:
11      'match[]':
12        #- '{job="prometheus"}'
13        #- '{__name__=~"job:.*"}'
14        - '{__name__=~"node_.*|container_.*"}'
15        - '{__name__=~"windows_.*"}'
16        - '{__name__=~"probe_.*"}'
17        #- '{job!=""}'
18        #- '{job="node"}'
19        #- '{job="blackbox_http_2xx"}'
20        #- '{job="blackbox_icmp"}'
21        #match[]={__name__=~"..*"}
22        #match[]="{__name__=~".+"}"
23    static_configs:
24      - targets:
25        - 'x.x.x.x:9090'
26    basic_auth:
27      username: '********'
28      password: '********'

Sanity Check: You can connect to http://x.x.x.x:9090/targets and find the federation job status.

Prometheus C pushes (forwards) metrics to Prometheus A

In this scenario, you first need to enable the Remote Write Receiver feature in Prometheus A. You just need to add --web.enable-remote-write-receiver to your docker-compose.yml, no need to modify prometheus.yml 

 1# prometheus.yml of A
 2  prometheus:
 3    image: prom/prometheus:latest
 4    container_name: prometheus
 5    expose:
 6    - 9090
 7    ports:
 8    - 9090:9090
 9    volumes:
10    - /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
11    - /opt/prometheus/web.yml:/etc/prometheus/web.yml:ro
12    command:
13    - '--config.file=/etc/prometheus/prometheus.yml'
14    - '--web.config.file=/etc/prometheus/web.yml'
15    - '--web.enable-remote-write-receiver'

Then, you need to modify docker-compose.yml and *prometheus.yml of Prometheus C. First, you need to enable Prometheus agent mode, just need to add --enable-feature=agent to docker-compose.yml. 

 1# docker-compose.yml of C
 2  prometheus:
 3    image: prom/prometheus:latest
 4    container_name: prometheus
 5    expose:
 6    - 9090
 7    ports:
 8    - 9090:9090
 9    volumes:
10    - /opt/grafana/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
11    command:
12    - '--config.file=/etc/prometheus/prometheus.yml'
13    - '--enable-feature=agent'
14    - '--log.level=debug'

Second, you need to edit prometheus.yml, add the remote_write section, and enter the URL of Prometheus A in it. 

1# prometheus.yml of C
2global:
3  scrape_interval: 15s
4
5remote_write:
6  - url: 'http://(Domain of Prometheus A):9090/api/v1/write'
7    basic_auth:
8      username: '******'
9      password: '******'

Sanity check: you can enable the log to debug level and check that it's sending the data. 

1prometheus           | ts=2022-08-31T06:58:46.065Z caller=dedupe.go:112 component=remote level=debug remote_name=3c577a url=http://x.x.x.x:3128/api/v1/write msg=QueueManager.calculateDesiredShards dataInRate=95.10224000000001 dataOutRate=273.83119999999997 dataKeptRatio=1 dataPendingRate=-178.72895999999997 dataPending=285.30672000000004 dataOutDuration=0.026267519444000003 timePerSample=9.592595527463637e-05 desiredShards=0.010491189203871395 highestSent=1.66192912e+09 highestRecv=1.661929123e+09

Note

You should enable authentication in Prometheus, you can refer to this page to hash your password https://prometheus.io/docs/guides/basic-auth/, and save it in your web.yml. 

1basic_auth_users:
2    username: ******************************************

Port forward

If you have a restricted firewall, you can use socat to forward the request to another host. 

1socat -v TCP-LISTEN:3128,fork,reuseaddr TCP:x.x.x.x:9090