Skip to main content

Working with Time Series Data

Now that you've built your first dashboard, let's dive deeper into the most common type of data you'll work with in Grafana: time series data. Understanding how to effectively handle time series is crucial for monitoring and observability.

Learning Goals

  • Understand what time series data is and why it's important
  • Learn to query and transform time series data effectively
  • Master time range selection and relative time queries
  • Handle gaps and null values in time series
  • Apply time series functions for meaningful analysis

What is Time Series Data?​

Time series data consists of data points indexed in time order. Each data point typically includes:

  • A timestamp
  • One or more numeric values
  • Optional tags or labels for categorization
Example time series data structure
http_requests_total{method="GET", status="200"} 1500 @ 1609459200
http_requests_total{method="POST", status="201"} 300 @ 1609459200
http_requests_total{method="GET", status="200"} 1520 @ 1609459260

In monitoring contexts, time series data helps you track metrics over time, identify trends, and detect anomalies.

Querying Time Series Data​

Basic Time Series Queries​

Let's start with practical examples using PromQL (Prometheus Query Language), one of the most common time series query languages:

Simple rate calculation
rate(http_requests_total[5m])

This query calculates the per-second average rate of HTTP requests over 5-minute windows.

Filtering by labels
rate(http_requests_total{method="GET", status=~"2.."}[5m])

This filters for GET requests with 2xx status codes using regular expression matching.

Working with Multiple Series​

Aggregating across dimensions
sum by (method) (rate(http_requests_total[5m]))

This aggregates request rates by HTTP method, giving you a separate time series for each method.

tip

Use by for grouping and without for excluding dimensions. Choose the approach that makes your visualization clearer!

Time Range and Resolution​

Setting Time Ranges​

Grafana provides flexible time range selection:

  • Absolute time: Specific start and end dates
  • Relative time: "Last 1 hour", "Last 7 days"
  • Quick ranges: Predefined common intervals
Using relative time in queries
rate(http_requests_total[1h])  // Uses dashboard time range
rate(http_requests_total[$__range])  // Uses Grafana's range variable

Understanding Resolution​

The $__interval variable represents the time between data points based on your graph width and time range:

Using interval for appropriate aggregation
sum(rate(http_requests_total[$__interval]))
warning

Too small intervals can cause performance issues, while too large intervals may hide important details. Let Grafana calculate optimal intervals when possible.

Handling Data Gaps and Null Values​

Time series data often has gaps. Here's how to handle them:

Gap Filling Strategies​

Forward-filling gaps
avg_over_time(temperature_celsius[10m]) or temperature_celsius
Using default values for missing data
avg_over_time(temperature_celsius[10m]) or vector(0)

Dealing with Sparse Data​

# For sparse counters, use rate with appropriate ranges
rate(sparse_counter_total[10m])

Time Series Functions and Transformations​

Common Mathematical Operations​

Calculating error rates
rate(http_requests_total{status=~"5.."}[5m]) 
/ 
rate(http_requests_total[5m]) * 100
Moving averages for smoothing
avg_over_time(rate(http_requests_total[5m])[10m:1m])

Time Shifting for Comparison​

Comparing current vs. previous period
rate(http_requests_total[5m])
/
rate(http_requests_total[5m] offset 1w)

This shows how current request rates compare to the same time last week.

Visualization Best Practices for Time Series​

Choosing the Right Graph Type​

  • Time series graphs: For continuous metrics
  • Stat panels: For current values or single numbers
  • Bar gauges: For showing values against thresholds
  • Heatmaps: For distribution over time

Color and Legend Configuration​

Configure your time series to make patterns immediately visible:

Effective color scheme example
- series: "high latency"
  color: "red"
- series: "medium latency" 
  color: "orange"
- series: "low latency"
  color: "green"

Common Pitfalls​

  • Ignoring data cardinality: Too many unique time series can overwhelm your database
  • Misunderstanding rate(): Remember it's per-second rate, not total counts
  • Wrong aggregation intervals: Too coarse hides details, too fine creates noise
  • Forgetting time zones: Ensure consistent timezone handling across data sources
  • Overlooking data retention: Old data might be aggregated or deleted based on retention policies

Summary​

Time series data is the foundation of Grafana dashboards. You've learned to query time series effectively, handle common issues like gaps and null values, apply useful transformations, and visualize time-based patterns clearly. Remember that good time series analysis balances detail with performance and always considers the time context of your data.

Quiz​

Show quiz
  1. What does the rate() function in PromQL calculate?
  2. Why is the $__interval variable important in time series queries?
  3. How can you compare current metrics with the same time last week?
  4. What's a common strategy for handling gaps in time series data?
  5. Why might having too many unique time series be problematic?

Answers

  1. rate() calculates the per-second average rate of increase for counter metrics over a specified time window
  2. $__interval represents the optimal time between data points based on graph width and time range, helping balance detail and performance
  3. Use the offset modifier: metric_name[range] offset 1w
  4. Use functions like or with default values, forward-filling, or appropriate aggregation to handle null values
  5. High cardinality (too many unique series) can overwhelm databases, causing performance issues and increased resource usage