Index Patterns and Data Views

Index patterns (renamed to "data views" in Kibana 8.x) define how Kibana connects to Elasticsearch indices. Every search, visualization, and dashboard depends on a properly configured data view.

What is a Data View?

A data view tells Kibana:

1. Which indices to query (via a pattern like logs-*)
2. Which field is the timestamp (for time-based analysis)
3. What fields are available (names, types, formats)

Elasticsearch Indices Data View Kibana Apps
┌──────────────────┐
│ logs-2024-01-01 │
│ logs-2024-01-02 ├──→ logs-* ──→ Discover, Dashboards,
│ logs-2024-01-03 │ (@timestamp) Visualizations, Alerts
│ logs-2024-01-04 │
└──────────────────┘

Without a data view, Kibana has no way to access your Elasticsearch data.

Data Views vs Index Patterns

The terminology changed in Kibana 8.x:

The functionality is the same. This guide uses "data view" but the concepts apply to both.

Creating a Data View

Via the UI

1. Go to Stack Management → Kibana → Data Views
2. Click "Create data view"
3. Fill in the form:

Name: Web Server Logs
Index pattern: webserver-logs-*
Timestamp field: @timestamp

4. Click "Save data view to Kibana"

Choosing an Index Pattern

The index pattern determines which Elasticsearch indices are included:

Wildcards in Patterns

* Matches everything
logs-* Matches any index starting with "logs-"
*-prod-* Matches any index containing "-prod-"
logs-*,-logs-debug-* Match logs-*, exclude logs-debug-*

Exclusion syntax: Use a comma followed by a minus sign to exclude indices:

Index pattern: logs-*,-logs-internal-*,-logs-debug-*
Matches: logs-web-*, logs-api-*, etc.
Excludes: logs-internal-*, logs-debug-*

Selecting a Timestamp Field

The timestamp field enables time-based filtering (the time picker):

Common timestamp fields:
- @timestamp (default for Beats, Logstash)
- timestamp (custom pipelines)
- order_date (business data)
- event.created (ECS format)
- event.ingested (when Elasticsearch received it)

No timestamp: Select "I don't want to use the time filter" for data without a time component (reference tables, configuration data).

Via API

Create data views programmatically:

# Create a data view
curl -X POST "localhost:5601/api/data_views/data_view" \
 -H "kbn-xsrf: true" \
 -H "Content-Type: application/json" \
 -d '{
 "data_view": {
 "title": "filebeat-*",
 "timeFieldName": "@timestamp",
 "name": "Filebeat Logs"
 }
 }'

# Response
{
 "data_view": {
 "id": "abc123-def456",
 "title": "filebeat-*",
 "timeFieldName": "@timestamp",
 "name": "Filebeat Logs",
 "fields": { ... }
 }
}

# List all data views
curl -X GET "localhost:5601/api/data_views" \
 -H "kbn-xsrf: true"

# Get a specific data view
curl -X GET "localhost:5601/api/data_views/data_view/abc123-def456" \
 -H "kbn-xsrf: true"

# Delete a data view
curl -X DELETE "localhost:5601/api/data_views/data_view/abc123-def456" \
 -H "kbn-xsrf: true"

Understanding Fields

Field Types

Each field in a data view has a type determined by its Elasticsearch mapping:

Field Properties

Common confusion: text fields are searchable but NOT aggregatable. Use the .keyword sub-field for aggregations.

Field: category (text) → Searchable, NOT aggregatable
Field: category.keyword → Searchable AND aggregatable

Viewing Field Details

1. Go to Stack Management → Data Views
2. Click a data view
3. See the full field list with:

• Field name
• Type
• Format
• Searchable/Aggregatable status
• Popularity (usage count)

Field Formatting

Customize how field values display in Kibana.

Setting a Field Format

1. Open data view → click field name → "Edit"
2. Choose a format:

Number Format Examples

Field: taxful_total_price
Format: Number
Pattern: $0,0.00

Result: 1234.5 → $1,234.50

Field: bytes
Format: Bytes
Pattern: 0.0 b

Result: 1536000 → 1.5 MB

URL Format

Turn field values into clickable links:

Field: order_id
Format: URL
Template: https://orders.example.com/{{value}}

Result: "12345" → clickable link to https://orders.example.com/12345

Field: username
Format: URL
Type: Image
Template: https://avatars.example.com/{{value}}.png

Result: Shows user avatar image inline

Color Format

Apply colors based on value ranges:

Field: response
Format: Color

Rules:
- Range 200-299: Background green, text white
- Range 300-399: Background yellow, text black
- Range 400-499: Background orange, text white
- Range 500-599: Background red, text white

Scripted Fields

Scripted fields compute values at query time using Painless (Elasticsearch's scripting language).

Creating a Scripted Field

1. Open data view → "Scripted fields" tab → "Add scripted field"
2. Configure:

Name: price_with_tax
Language: painless
Type: number
Format: Number ($0,0.00)
Script: doc['price'].value * 1.1

Common Scripted Field Examples

Calculate a derived value:

// Full name from first and last
Name: full_name
Type: string
Script:
 doc['first_name.keyword'].value + ' ' + doc['last_name.keyword'].value

Extract hour from timestamp:

// Hour of day (0-23) for time-of-day analysis
Name: hour_of_day
Type: number
Script:
 doc['@timestamp'].value.getHour()

Categorize numeric values:

// Price tier
Name: price_tier
Type: string
Script:
 if (doc['price'].value < 25) return 'Budget';
 if (doc['price'].value < 100) return 'Standard';
 return 'Premium';

Calculate percentage:

// Error rate
Name: error_rate_pct
Type: number
Script:
 (doc['error_count'].value / doc['total_count'].value) * 100

Scripted Field Limitations

 Slow on large datasets (computed per document at query time)
 Cannot be used in filters (only in visualizations and tables)
 No access to _source (only doc values)
 May break if underlying fields change

 Good for simple calculations
 Useful for prototyping before creating an ingest pipeline
 Works without reindexing data

Better alternatives:

• Runtime fields (Kibana 8.x) - more flexible, better performance
• Ingest pipeline - compute at index time, stored in document
• Transform - pre-aggregate data into summary indices

Runtime Fields

Runtime fields are the modern replacement for scripted fields in Kibana 8.x.

Creating a Runtime Field

1. Open data view → "Add field" (or click the + icon)
2. Toggle "Set value" on
3. Choose type and write the script:

Name: response_category
Type: Keyword

Script (Painless):
 if (doc['response'].value >= 200 && doc['response'].value < 300) {
 emit('success');
 } else if (doc['response'].value >= 400 && doc['response'].value < 500) {
 emit('client_error');
 } else if (doc['response'].value >= 500) {
 emit('server_error');
 } else {
 emit('other');
 }

Runtime Field Advantages over Scripted Fields

Runtime Field Examples

Mask sensitive data:

Name: masked_email
Type: Keyword
Script:
 def email = doc['email.keyword'].value;
 def parts = email.splitOnToken('@');
 emit(parts[0].substring(0, 2) + '***@' + parts[1]);
// "john.doe@example.com" → "jo***@example.com"

Extract URL path segments:

Name: api_version
Type: Keyword
Script:
 def path = doc['url.path.keyword'].value;
 if (path.startsWith('/api/v')) {
 emit(path.substring(5, 7));
 } else {
 emit('unknown');
 }
// "/api/v2/users" → "v2"

Managing Multiple Data Views

Data View Strategy

Organize data views by purpose:

Production Monitoring:
 ├── logs-prod-* (all production logs)
 ├── metrics-prod-* (all production metrics)
 └── apm-prod-* (production APM data)

Development:
 ├── logs-dev-* (development logs)
 └── metrics-dev-* (development metrics)

Business Analytics:
 ├── orders-* (all order data)
 ├── customers-* (customer records)
 └── products-* (product catalog)

Security:
 ├── auditbeat-* (audit logs)
 ├── packetbeat-* (network data)
 └── winlogbeat-* (Windows events)

Cross-Cluster Data Views

If you have multiple Elasticsearch clusters connected via cross-cluster search:

Index pattern: cluster_name:logs-*

Example:
 us-east:logs-* (logs from US East cluster)
 eu-west:logs-* (logs from EU West cluster)
 *:logs-* (logs from all clusters)

Setting a Default Data View

1. Go to Stack Management → Advanced Settings
2. Search for defaultIndex
3. Select the data view ID to use as default
4. Click "Save"

The default data view is pre-selected when opening Discover or creating visualizations.

Refreshing Data Views

When index mappings change (new fields added, types updated), refresh the data view:

Manual Refresh

1. Go to Stack Management → Data Views
2. Open the data view
3. Click the refresh icon () in the top-right

When to Refresh

Refresh when:
 New fields appear in your data
 Field types change (rare, usually a reindex)
 New indices matching the pattern are created
 Fields show as "unknown" type

No refresh needed:
 New documents added to existing indices
 Time range changes
 Query changes

Data View Spaces

Data views are scoped to Kibana Spaces:

Default Space:
 ├── logs-*
 ├── metrics-*
 └── apm-*

Marketing Space:
 ├── analytics-*
 └── campaigns-*

Security Space:
 ├── auditbeat-*
 └── security-*

To share a data view across spaces, create it in each space or use the Saved Objects copy feature:

1. Stack Management → Saved Objects
2. Find the data view
3. Click "Copy to space"
4. Select target spaces

Practical Examples

Example 1: Multi-Environment Setup

# Create data views for each environment via API
for ENV in prod staging dev; do
 curl -X POST "localhost:5601/api/data_views/data_view" \
 -H "kbn-xsrf: true" \
 -H "Content-Type: application/json" \
 -d "{
 \"data_view\": {
 \"title\": \"logs-${ENV}-*\",
 \"timeFieldName\": \"@timestamp\",
 \"name\": \"Logs (${ENV})\"
 }
 }"
done

Example 2: Data View with Field Formatting

After creating the data view, set up useful field formats:

Field: response → Color format (green 2xx, red 5xx)
Field: bytes → Bytes format (auto KB/MB/GB)
Field: url.full → URL format (clickable link)
Field: duration_ms → Duration format (humanized)
Field: price → Number format ($0,0.00)

Example 3: Runtime Field for SLA Tracking

// Create a runtime field: sla_status
Name: sla_status
Type: Keyword
Script:
 def responseTime = doc['response_time_ms'].value;
 if (responseTime <= 200) {
 emit('within_sla');
 } else if (responseTime <= 500) {
 emit('warning');
 } else {
 emit('sla_breach');
 }

Now use this in visualizations:

• Pie chart: SLA compliance breakdown
• Metric: Percentage within SLA
• Alert: Trigger when sla_breach count exceeds threshold

Common Issues

"No matching indices"

Cause: No Elasticsearch indices match the pattern.

Fix:

1. Verify indices exist: GET _cat/indices/logs-*
2. Check for typos in the pattern
3. Confirm the index name format matches your pattern

"Field not found" in queries

Cause: Field exists in some indices but not others matched by the pattern.

Fix:

1. Refresh the data view
2. Check which indices actually contain the field
3. Narrow the index pattern if needed

Conflicting field types

Cause: Same field name has different types across indices (e.g., status is keyword in one index and long in another).

Fix:

1. Kibana shows a conflict warning icon
2. Choose the correct type for your use case
3. Better: fix the mapping inconsistency at the source with an index template

# Check field mappings across indices
curl "localhost:9200/logs-*/_mapping/field/status"

Too many fields

Cause: Data view matches indices with thousands of fields, slowing down Kibana.

Fix:

1. Narrow the index pattern to fewer indices
2. Use metaFields setting to limit overhead
3. Consider field filtering in the data view

# kibana.yml - limit fields
# (use with caution, may hide needed fields)
xpack.discoverEnhanced.actions.exploreDataInContextMenu.enabled: true

Next Steps

Continue to 08-lens.md for building drag-and-drop visualizations with Lens.