Index Patterns and Data Views

Index patterns (renamed to "data views" in Kibana 8.x) define how Kibana connects to Elasticsearch indices. Every search, visualization, and dashboard depends on a properly configured data view.

What is a Data View?

A data view tells Kibana:

1. Which indices to query (via a pattern like logs-*)
2. Which field is the timestamp (for time-based analysis)
3. What fields are available (names, types, formats)

Elasticsearch Indices           Data View              Kibana Apps
┌──────────────────┐
│ logs-2024-01-01  │
│ logs-2024-01-02  ├──→  logs-*  ──→  Discover, Dashboards,
│ logs-2024-01-03  │     (@timestamp)     Visualizations, Alerts
│ logs-2024-01-04  │
└──────────────────┘

Without a data view, Kibana has no way to access your Elasticsearch data.

Data Views vs Index Patterns

The terminology changed in Kibana 8.x:

The functionality is the same. This guide uses "data view" but the concepts apply to both.

Creating a Data View

Via the UI

1. Go to Stack Management → Kibana → Data Views
2. Click "Create data view"
3. Fill in the form:

Name:           Web Server Logs
Index pattern:  webserver-logs-*
Timestamp field: @timestamp

4. Click "Save data view to Kibana"

Choosing an Index Pattern

The index pattern determines which Elasticsearch indices are included:

Wildcards in Patterns

*           Matches everything
logs-*      Matches any index starting with "logs-"
*-prod-*    Matches any index containing "-prod-"
logs-*,-logs-debug-*    Match logs-*, exclude logs-debug-*

Exclusion syntax: Use a comma followed by a minus sign to exclude indices:

Index pattern: logs-*,-logs-internal-*,-logs-debug-*
Matches: logs-web-*, logs-api-*, etc.
Excludes: logs-internal-*, logs-debug-*

Selecting a Timestamp Field

The timestamp field enables time-based filtering (the time picker):

Common timestamp fields:
- @timestamp          (default for Beats, Logstash)
- timestamp           (custom pipelines)
- order_date          (business data)
- event.created       (ECS format)
- event.ingested      (when Elasticsearch received it)

No timestamp: Select "I don't want to use the time filter" for data without a time component (reference tables, configuration data).

Via API

Create data views programmatically:

# Create a data view
curl -X POST "localhost:5601/api/data_views/data_view" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -d '{
    "data_view": {
      "title": "filebeat-*",
      "timeFieldName": "@timestamp",
      "name": "Filebeat Logs"
    }
  }'

# Response
{
  "data_view": {
    "id": "abc123-def456",
    "title": "filebeat-*",
    "timeFieldName": "@timestamp",
    "name": "Filebeat Logs",
    "fields": { ... }
  }
}

# List all data views
curl -X GET "localhost:5601/api/data_views" \
  -H "kbn-xsrf: true"

# Get a specific data view
curl -X GET "localhost:5601/api/data_views/data_view/abc123-def456" \
  -H "kbn-xsrf: true"

# Delete a data view
curl -X DELETE "localhost:5601/api/data_views/data_view/abc123-def456" \
  -H "kbn-xsrf: true"

Understanding Fields

Field Types

Each field in a data view has a type determined by its Elasticsearch mapping:

Field Properties

Common confusion: text fields are searchable but NOT aggregatable. Use the .keyword sub-field for aggregations.

Field: category (text)        → Searchable, NOT aggregatable
Field: category.keyword       → Searchable AND aggregatable

Viewing Field Details

1. Go to Stack Management → Data Views
2. Click a data view
3. See the full field list with:
   • Field name
   • Type
   • Format
   • Searchable/Aggregatable status
   • Popularity (usage count)

Field Formatting

Customize how field values display in Kibana.

Setting a Field Format

1. Open data view → click field name → "Edit"
2. Choose a format:

Number Format Examples

Field: taxful_total_price
Format: Number
Pattern: $0,0.00

Result: 1234.5 → $1,234.50

Field: bytes
Format: Bytes
Pattern: 0.0 b

Result: 1536000 → 1.5 MB

URL Format

Turn field values into clickable links:

Field: order_id
Format: URL
Template: https://orders.example.com/{{value}}

Result: "12345" → clickable link to https://orders.example.com/12345

Field: username
Format: URL
Type: Image
Template: https://avatars.example.com/{{value}}.png

Result: Shows user avatar image inline

Color Format

Apply colors based on value ranges:

Field: response
Format: Color

Rules:
- Range 200-299: Background green, text white
- Range 300-399: Background yellow, text black
- Range 400-499: Background orange, text white
- Range 500-599: Background red, text white

Scripted Fields

Scripted fields compute values at query time using Painless (Elasticsearch's scripting language).

Creating a Scripted Field

1. Open data view → "Scripted fields" tab → "Add scripted field"
2. Configure:

Name: price_with_tax
Language: painless
Type: number
Format: Number ($0,0.00)
Script: doc['price'].value * 1.1

Common Scripted Field Examples

Calculate a derived value:

// Full name from first and last
Name: full_name
Type: string
Script:
  doc['first_name.keyword'].value + ' ' + doc['last_name.keyword'].value

Extract hour from timestamp:

// Hour of day (0-23) for time-of-day analysis
Name: hour_of_day
Type: number
Script:
  doc['@timestamp'].value.getHour()

Categorize numeric values:

// Price tier
Name: price_tier
Type: string
Script:
  if (doc['price'].value < 25) return 'Budget';
  if (doc['price'].value < 100) return 'Standard';
  return 'Premium';

Calculate percentage:

// Error rate
Name: error_rate_pct
Type: number
Script:
  (doc['error_count'].value / doc['total_count'].value) * 100

Scripted Field Limitations

❌ Slow on large datasets (computed per document at query time)
❌ Cannot be used in filters (only in visualizations and tables)
❌ No access to _source (only doc values)
❌ May break if underlying fields change

✅ Good for simple calculations
✅ Useful for prototyping before creating an ingest pipeline
✅ Works without reindexing data

Better alternatives:

• Runtime fields (Kibana 8.x) - more flexible, better performance
• Ingest pipeline - compute at index time, stored in document
• Transform - pre-aggregate data into summary indices

Runtime Fields

Runtime fields are the modern replacement for scripted fields in Kibana 8.x.

Creating a Runtime Field

1. Open data view → "Add field" (or click the + icon)
2. Toggle "Set value" on
3. Choose type and write the script:

Name: response_category
Type: Keyword

Script (Painless):
  if (doc['response'].value >= 200 && doc['response'].value < 300) {
    emit('success');
  } else if (doc['response'].value >= 400 && doc['response'].value < 500) {
    emit('client_error');
  } else if (doc['response'].value >= 500) {
    emit('server_error');
  } else {
    emit('other');
  }

Runtime Field Advantages over Scripted Fields

Runtime Field Examples

Mask sensitive data:

Name: masked_email
Type: Keyword
Script:
  def email = doc['email.keyword'].value;
  def parts = email.splitOnToken('@');
  emit(parts[0].substring(0, 2) + '***@' + parts[1]);
// "john.doe@example.com" → "jo***@example.com"

Extract URL path segments:

Name: api_version
Type: Keyword
Script:
  def path = doc['url.path.keyword'].value;
  if (path.startsWith('/api/v')) {
    emit(path.substring(5, 7));
  } else {
    emit('unknown');
  }
// "/api/v2/users" → "v2"

Managing Multiple Data Views

Data View Strategy

Organize data views by purpose:

Production Monitoring:
  ├── logs-prod-*           (all production logs)
  ├── metrics-prod-*        (all production metrics)
  └── apm-prod-*            (production APM data)

Development:
  ├── logs-dev-*            (development logs)
  └── metrics-dev-*         (development metrics)

Business Analytics:
  ├── orders-*              (all order data)
  ├── customers-*           (customer records)
  └── products-*            (product catalog)

Security:
  ├── auditbeat-*           (audit logs)
  ├── packetbeat-*          (network data)
  └── winlogbeat-*          (Windows events)

Cross-Cluster Data Views

If you have multiple Elasticsearch clusters connected via cross-cluster search:

Index pattern: cluster_name:logs-*

Example:
  us-east:logs-*            (logs from US East cluster)
  eu-west:logs-*            (logs from EU West cluster)
  *:logs-*                  (logs from all clusters)

Setting a Default Data View

1. Go to Stack Management → Advanced Settings
2. Search for defaultIndex
3. Select the data view ID to use as default
4. Click "Save"

The default data view is pre-selected when opening Discover or creating visualizations.

Refreshing Data Views

When index mappings change (new fields added, types updated), refresh the data view:

Manual Refresh

1. Go to Stack Management → Data Views
2. Open the data view
3. Click the refresh icon (🔄) in the top-right

When to Refresh

Refresh when:
✅ New fields appear in your data
✅ Field types change (rare, usually a reindex)
✅ New indices matching the pattern are created
✅ Fields show as "unknown" type

No refresh needed:
❌ New documents added to existing indices
❌ Time range changes
❌ Query changes

Data View Spaces

Data views are scoped to Kibana Spaces:

Default Space:
  ├── logs-*
  ├── metrics-*
  └── apm-*

Marketing Space:
  ├── analytics-*
  └── campaigns-*

Security Space:
  ├── auditbeat-*
  └── security-*

To share a data view across spaces, create it in each space or use the Saved Objects copy feature:

1. Stack Management → Saved Objects
2. Find the data view
3. Click "Copy to space"
4. Select target spaces

Practical Examples

Example 1: Multi-Environment Setup

# Create data views for each environment via API
for ENV in prod staging dev; do
  curl -X POST "localhost:5601/api/data_views/data_view" \
    -H "kbn-xsrf: true" \
    -H "Content-Type: application/json" \
    -d "{
      \"data_view\": {
        \"title\": \"logs-${ENV}-*\",
        \"timeFieldName\": \"@timestamp\",
        \"name\": \"Logs (${ENV})\"
      }
    }"
done

Example 2: Data View with Field Formatting

After creating the data view, set up useful field formats:

Field: response         → Color format (green 2xx, red 5xx)
Field: bytes            → Bytes format (auto KB/MB/GB)
Field: url.full         → URL format (clickable link)
Field: duration_ms      → Duration format (humanized)
Field: price            → Number format ($0,0.00)

Example 3: Runtime Field for SLA Tracking

// Create a runtime field: sla_status
Name: sla_status
Type: Keyword
Script:
  def responseTime = doc['response_time_ms'].value;
  if (responseTime <= 200) {
    emit('within_sla');
  } else if (responseTime <= 500) {
    emit('warning');
  } else {
    emit('sla_breach');
  }

Now use this in visualizations:

• Pie chart: SLA compliance breakdown
• Metric: Percentage within SLA
• Alert: Trigger when sla_breach count exceeds threshold

Common Issues

"No matching indices"

Cause: No Elasticsearch indices match the pattern.

Fix:

1. Verify indices exist: GET _cat/indices/logs-*
2. Check for typos in the pattern
3. Confirm the index name format matches your pattern

"Field not found" in queries

Cause: Field exists in some indices but not others matched by the pattern.

Fix:

1. Refresh the data view
2. Check which indices actually contain the field
3. Narrow the index pattern if needed

Conflicting field types

Cause: Same field name has different types across indices (e.g., status is keyword in one index and long in another).

Fix:

1. Kibana shows a conflict warning icon
2. Choose the correct type for your use case
3. Better: fix the mapping inconsistency at the source with an index template

# Check field mappings across indices
curl "localhost:9200/logs-*/_mapping/field/status"

Too many fields

Cause: Data view matches indices with thousands of fields, slowing down Kibana.

Fix:

1. Narrow the index pattern to fewer indices
2. Use metaFields setting to limit overhead
3. Consider field filtering in the data view

# kibana.yml - limit fields
# (use with caution, may hide needed fields)
xpack.discoverEnhanced.actions.exploreDataInContextMenu.enabled: true

Summary

In this chapter, you learned:

• ✅ What data views are and why they matter
• ✅ Creating data views with index patterns and timestamp fields
• ✅ Understanding field types and their implications
• ✅ Formatting fields for better readability
• ✅ Using scripted fields and runtime fields for computed values
• ✅ Managing data views across environments and spaces
• ✅ Programmatic data view creation via API
• ✅ Troubleshooting common data view issues

Next: Building drag-and-drop visualizations with Lens!