Docker Logs Observability Using Grafana

Managing logs is a developer’s responsibility—never be caught off guard when asked, “Where’s the error?” 😆 If you’re managing a single-instance container on a VPS, this article is for you. We’ll implement a simple yet effective solution that eliminates the need to log in via SSH just to check logs. With this setup, you can go beyond just viewing logs—you’ll be able to create dashboards, track metrics, perform searches, and even set up alerts.

Advantages of This Approach

With this setup, you don’t have to worry about managing log files, retention, or storage on your application. Simply log to the console, and Promtail will automatically collect and send the logs to Loki. But remember, you have to follow logging best practices to maximize the benefits of this observability stack:

• Use appropriate logging levels (INFO, WARN, ERROR, etc.).
• Write structured and meaningful log messages.
• Keep logs concise, avoid excessive logging to prevent noise.
• Never expose sensitive information in logs.

Why Use Grafana?

Grafana is a powerful open-source tool for monitoring, visualizing, and analyzing logs and metrics. It allows you to turn raw log data into meaningful insights with interactive dashboards. Unlike traditional log viewing methods that require manual SSH access, Grafana provides a centralized interface to track system performance and detect anomalies in real-time.

However, Grafana alone is not enough for log observability. We need a complete stack that includes:

• Grafana: The visualization and dashboard tool that helps you explore log data and set up alerts.
• Loki: A log aggregation system designed for efficiency and scalability, acting as a backend for storing and indexing logs.
• Promtail: A lightweight log collector that ships logs from Docker containers to Loki.

All three components can run on Docker. In the next section, I’ll provide a simple Docker Compose setup to deploy them easily.

But before that, you need to configure Loki and Promtail. Here is the configuration for Loki.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: debug
  grpc_server_max_concurrent_streams: 1000

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

limits_config:
  metric_aggregation_enabled: true

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

pattern_ingester:
  enabled: true
  metric_aggregation:
    loki_address: localhost:3100

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

Save this configuration somewhere on your server. Next, create a Promtail configuration to collect Docker logs and send them to Loki.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker-logs
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_image]
        target_label: image

Next, create the Docker Compose configuration.

version: "3.3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_FEATURE_TOGGLES_ENABLE=alertingSimplifiedRouting,alertingQueryAndExpressionsStepMode
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

Fire up all three services with the following command:

docker-compose up -d

You can verify that Loki is up and running.

• To view readiness, navigate to http://your-server-ip:3100/ready.
• To view metrics, navigate to http://your-server-ip:3100/metrics.

Next, you can access Grafana by opening http://your-server-ip:3000 in your browser. By default, the login credentials are:

• Username: admin
• Password: admin

Upon your first login, Grafana will prompt you to change the password for security reasons. Once inside, you can start adding Loki as a data source and explore your logs in a more interactive way. 🚀