Key Concepts

• Scraping
• PromQL
• Alert rules

Concept Deep Dive

Scraping

Scraping is a core idea you’ll use repeatedly while working with Prometheus.

Why it matters: Understanding Scraping helps you design safer workflows and troubleshoot issues faster.

Practice:

• Explain Scraping in your own words (1 minute rule).
• Find where Scraping appears in real docs/configs for Prometheus.
• Create a small example that uses Scraping, then break it and fix it.

PromQL

PromQL is a core idea you’ll use repeatedly while working with Prometheus.

Why it matters: Understanding PromQL helps you design safer workflows and troubleshoot issues faster.

Practice:

• Explain PromQL in your own words (1 minute rule).
• Find where PromQL appears in real docs/configs for Prometheus.
• Create a small example that uses PromQL, then break it and fix it.

Alert rules

Alert rules is a core idea you’ll use repeatedly while working with Prometheus.

Why it matters: Understanding Alert rules helps you design safer workflows and troubleshoot issues faster.

Practice:

• Explain Alert rules in your own words (1 minute rule).
• Find where Alert rules appears in real docs/configs for Prometheus.
• Create a small example that uses Alert rules, then break it and fix it.

Core Workflow

1. Foundations

Start with core Prometheus concepts and basic setup so you can use it safely in day-to-day work.

Goals:

• Understand Prometheus fundamentals
• Set up local/dev environment
• Run first working example

2. Team Workflow

Integrate Prometheus into real team practices with repeatable conventions and collaboration patterns.

Goals:

• Adopt standards and naming conventions
• Integrate with repositories and CI/CD
• Create reusable templates

3. Production Operations

Use Prometheus in production with observability, security, and rollback plans.

Goals:

• Monitor behavior and failures
• Secure access and secrets
• Define incident and rollback flow

4. Scale and Optimization

Continuously improve reliability, performance, and cost while standardizing usage across services.

Goals:

• Improve performance and cost
• Automate compliance checks
• Document best practices for the team

Quick Start

1. Run Prometheus

docker run -p 9090:9090 prom/prometheus

1. Open UI

http://localhost:9090

1. Run query

up

Tutorial Series

A tutorial-style sequence (like a handbook). Do these in order to build skill from beginner to production.

Tutorial 1: Visibility First

Goal: Create signals that help you debug incidents faster.

Steps:

1. Verify you understand what the tool does and what problem it solves.
2. Install or enable it on your machine (or in a sandbox environment).
3. Run the smallest working example and write down what happened.
4. Pick 3 golden signals: latency, traffic, errors (and saturation if possible).
5. Create a minimal dashboard and one actionable alert.

Checkpoints:

• You can answer: is it broken and who is impacted?
• Your alert is not noisy

Exercises:

• Write a runbook for your alert
• Add log correlation (request ID)

Tutorial 2: Reduce MTTR with Tracing

Goal: Make debugging cross-service requests simpler.

Steps:

1. Add a trace ID and propagate it through services.
2. Use traces to find the slow span.

Checkpoints:

• You can pinpoint the bottleneck
• You can reproduce a slow request

Exercises:

• Create an incident drill and write a short postmortem
• Tune thresholds based on real traffic

Command Cheatsheet

• promtool check config prometheus.yml: Validate Prometheus config.
• promtool check rules alerts.yml: Validate alert rule files.
• curl http://localhost:9090/-/ready: Check readiness endpoint.
• curl http://localhost:9090/-/healthy: Check health endpoint.
• curl http://localhost:9090/api/v1/targets: List scrape targets via API.
• curl http://localhost:9090/api/v1/rules: List active rules via API.
• curl http://localhost:9090/api/v1/alerts: List active alerts via API.
• up: PromQL: target is up (1) or down (0).
• rate(http_requests_total[5m]): PromQL: request rate over 5m.
• sum(rate(http_requests_total[5m])) by (status): PromQL: rate by status.
• histogram_quantile(0.95, sum(rate(request_duration_bucket[5m])) by (le)): PromQL: p95 latency from histogram.

Learning Path

• Metric collection
• Query writing
• Alert design

Beginner To Advanced Path

Beginner Path (Foundations)

What to learn:

• Learn Prometheus terminology and the “why” behind it
• Install/setup and run a first working example
• Understand the main components and the default workflow
• Learn safe debugging: where to look when something fails
• Build a small checklist for your own repeatable setup
• Write notes (commands, errors, fixes) while learning

Hands-on labs:

• Follow a hello-world style tutorial and document every step
• Break one config intentionally and fix it (learn error patterns)
• Write a 10-command cheat sheet you can reuse later
• Create a simple diagram of the tool’s flow in your own words

Milestones:

• You can explain the tool in 2 minutes
• You can reproduce a working setup from scratch
• You can troubleshoot the top 3 common failures
• You can share a clean quick-start with someone else

Intermediate Path (Real Workflows)

What to learn:

• Use the tool inside a realistic DevOps workflow
• Create reusable templates/configs and standard naming conventions
• Add security basics: secrets handling and least privilege
• Reduce toil: automate repeated steps and build confidence
• Make the workflow faster and safer (cache, validations, checks)
• Document the workflow as if onboarding a new teammate

Hands-on labs:

• Integrate it with a CI pipeline (lint/build/test/deploy style flow)
• Parameterize config for dev/stage/prod environments
• Create a runbook: steps to validate and roll back a change
• Add a preflight validation step that blocks unsafe changes

Milestones:

• You can onboard another person with your docs
• You can run the tool consistently across environments
• You can explain tradeoffs (speed vs safety, flexibility vs complexity)
• You can debug failures using logs/outputs without guesswork

Advanced Path (Production & Scale)

What to learn:

• Operate the tool safely in production with guardrails
• Add observability: metrics/logs/traces and meaningful alerts
• Optimize performance/cost and standardize across multiple services
• Design failure modes and recovery (rollback, restore, incident flow)
• Create upgrade strategy and test it (versioning, compatibility)
• Create ownership: docs, alerts, dashboards, and operational SLAs

Hands-on labs:

• Add policy checks (security scans, approvals, protected environments)
• Load test or scale test the workflow and measure bottlenecks
• Create an incident simulation and write a postmortem template
• Automate audits: drift checks, compliance checks, and reports

Milestones:

• You can detect failures quickly and recover safely
• You can maintain the setup long-term (upgrade strategy, docs, ownership)
• You can explain architecture decisions and alternatives
• You can standardize patterns across multiple services/teams

Hands-On Labs

Beginner Labs

• Install/setup and verify version
• Run the smallest working example
• Change one parameter and observe the behavior
• Cause a safe failure and document the fix

Intermediate Labs

• Integrate into a realistic workflow (pipeline, deploy, or automation)
• Parameterize configuration for two environments
• Add validation and rollback steps
• Write a runbook (steps + commands) for common failures

Advanced Labs

• Add guardrails (policy checks, approvals, least privilege)
• Add observability and meaningful alerts
• Load/scale test and identify bottlenecks
• Create an upgrade + rollback plan and test it

Advanced Topics

• High-cardinality control and cost management
• Alert fatigue reduction: symptoms vs causes
• Tracing strategy and sampling
• Dashboard design patterns for Prometheus
• Incident response: triage, mitigation, postmortem

Production Patterns

• Golden signals dashboards (latency, traffic, errors, saturation)
• Alert on symptoms, not noise (reduce false positives)
• Runbooks linked in alerts
• SLOs and error budgets to drive changes around Prometheus
• Log/metric retention policies and cost controls

Real-World Scenarios

• Use Prometheus to detect incidents with actionable alerts (not noisy ones).
• Reduce MTTR by correlating metrics, logs, and traces.
• Create dashboards that reflect user impact (latency, errors, saturation).

Troubleshooting

• Reproduce the issue with the smallest possible example
• Check logs/output first, then configuration, then permissions/credentials
• Validate inputs (versions, environment variables, file paths, network access)
• Rollback to last known-good state if production is affected
• Write down the root cause and add a guardrail so it does not repeat

Runbook Templates

Use these templates to make your docs feel like real production documentation.

Deploy Runbook

• Purpose
• Preconditions (secrets, access, approvals)
• Steps to deploy (exact commands)
• Post-deploy verification (health checks)
• Rollback steps
• Owner and escalation

Incident Triage Runbook

• Impact assessment (who is impacted?)
• Current signals (errors, latency, saturation)
• Recent changes (deploys, config, infra)
• First checks (logs, health endpoints, dependencies)
• Mitigation steps (rate limiting, rollback, scale)
• Follow-up actions (postmortem, guardrails)

Checklist (Copy/Paste)

• What changed since it last worked?
• What do logs say at the exact failure time?
• Is the service reachable on the expected port and DNS?
• Are credentials/permissions valid?
• Is disk full, memory exhausted, or CPU pegged?
• Do we have a safe rollback plan and is it tested?

Security & Best Practices

• Never hardcode secrets in code or commits
• Use least privilege (roles, scopes, minimal permissions)
• Prefer reproducible builds/configs over manual steps
• Add validations before applying changes (lint/validate/plan/dry-run)
• Keep documentation and runbooks updated
• Version pin critical dependencies and plan upgrades

Common Error Patterns

Symptom

Too many alerts and the team ignores them

Likely cause: Alerting on causes not symptoms; thresholds too sensitive

Fix steps:

• Alert on user impact (errors/latency) and page only on urgency
• Add runbooks and clear ownership
• Reduce noisy alerts and use dashboards for investigation

FAQ

What is Prometheus used for?

Prometheus is used to standardize and automate parts of delivery and operations so teams can ship faster and more reliably.

How long does it take to learn Prometheus?

You can get productive in days with fundamentals, but production mastery comes from building workflows, debugging failures, and operating it over time.

What should I learn before Prometheus?

Learn basic Linux + Git first, then follow the prerequisites section. Fundamentals make every advanced topic easier.

How do I use Prometheus safely in production?

Add guardrails: least privilege, validation before apply/deploy, monitoring, and a tested rollback plan.

Common Mistakes

• Using defaults in production without security hardening
• Skipping monitoring and post-deployment validation
• No rollback strategy for failed changes
• Over-complex setup before mastering fundamentals

Production Readiness Checklist

• Access control and least privilege applied
• Secrets managed securely
• Monitoring and alerting enabled
• Rollback and recovery process tested
• Documentation updated for team onboarding

Mini Projects

• Build a small project that uses Prometheus in a realistic workflow
• Write a checklist for production usage
• Create a troubleshooting runbook for common failures
• Create a one-page internal doc: setup, usage, debugging, rollback

Interview Questions

• Explain what Prometheus is and where it fits in DevOps.
• Describe a real problem you solved using Prometheus.
• What can go wrong in production, and how do you detect and recover?
• What is the difference between metrics, logs, and traces?
• How do you avoid alert fatigue?
• What are SLIs and SLOs, and how do they help reliability?

References

• Official documentation
• Video learning