Introduction to Kibana

What is Kibana?

Kibana is an open-source data visualization and exploration tool designed for Elasticsearch. It provides a user-friendly interface to:

• Search and explore your data interactively
• Visualize data with charts, graphs, and maps
• Build dashboards for real-time monitoring
• Manage the Elastic Stack
• Share insights with your team

Think of Kibana as the "window" into your Elasticsearch data - it makes complex data accessible and actionable.

The Elastic Stack (ELK Stack)

Kibana is one component of the Elastic Stack, formerly known as the ELK Stack:

How They Work Together

Data Sources → Beats/Logstash → Elasticsearch → Kibana → Users
    ↓              ↓                  ↓             ↓
  Logs          Transform           Index       Visualize
  Metrics        Enrich            Search       Dashboard
  Events         Filter             Store        Explore

Key Features

1. Discover

Explore your data with powerful search and filtering capabilities.

Search: status:200 AND method:GET
Result: Find all successful GET requests

2. Visualizations

Create various chart types:

• Line, area, and bar charts
• Pie charts and metrics
• Heat maps and tag clouds
• Geographic maps
• Gauges and goals

3. Dashboards

Combine multiple visualizations into interactive dashboards for a full view of your data.

4. Canvas

Design pixel-perfect presentations and infographics with your live data.

5. Machine Learning

Detect anomalies and forecast trends automatically.

6. Alerting

Set up rules to notify you when specific conditions are met.

7. APM (Application Performance Monitoring)

Monitor application performance, trace requests, and identify bottlenecks.

Common Use Cases

Log Analytics

Monitor application logs, identify errors, and track system health.

Use case: Finding errors in production
- Ingest logs via Filebeat
- Filter for error level messages
- Create dashboard showing error trends
- Set alerts for critical errors

Security Monitoring

Detect threats, investigate incidents, and maintain compliance.

Use case: Detecting suspicious login attempts
- Collect auth logs
- Visualize failed login attempts
- Create alerts for unusual patterns
- Use ML to detect anomalies

Business Analytics

Track KPIs, monitor user behavior, and make data-driven decisions.

Use case: E-commerce metrics
- Track sales in real-time
- Monitor conversion rates
- Visualize customer journey
- Compare performance over time

Infrastructure Monitoring

Track system performance, resource usage, and availability.

Use case: Server health monitoring
- Collect metrics via Metricbeat
- Dashboard for CPU, memory, disk
- Alert on threshold breaches
- Predict capacity needs

Installation

Option 1: Docker (Recommended for Learning)

# Create a network
docker network create elastic

# Start Elasticsearch
docker run -d \
  --name elasticsearch \
  --net elastic \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  docker.elastic.co/elasticsearch/elasticsearch:8.11.0

# Start Kibana
docker run -d \
  --name kibana \
  --net elastic \
  -p 5601:5601 \
  -e "ELASTICSEARCH_HOSTS=http://elasticsearch:9200" \
  docker.elastic.co/kibana/kibana:8.11.0

# Wait about 1 minute, then access Kibana at http://localhost:5601

Option 2: Native Installation (macOS)

# Install via Homebrew
brew tap elastic/tap
brew install elastic/tap/elasticsearch-full
brew install elastic/tap/kibana-full

# Start Elasticsearch
elasticsearch

# In another terminal, start Kibana
kibana

# Access at http://localhost:5601

Option 3: Native Installation (Linux - Debian/Ubuntu)

# Import Elastic GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

# Add repository
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Install Elasticsearch and Kibana
sudo apt-get update
sudo apt-get install elasticsearch kibana

# Start services
sudo systemctl start elasticsearch
sudo systemctl start kibana

# Enable on boot
sudo systemctl enable elasticsearch
sudo systemctl enable kibana

Option 4: Elastic Cloud (Production)

For production use, consider Elastic Cloud:

1. Sign up for a free trial
2. Create a deployment
3. Get your Kibana URL and credentials
4. Start building immediately

Pros: Managed, scalable, secure, automatic updates Cons: Costs money after trial

Verifying Installation

After installation, verify everything works:

# Check Elasticsearch is running
curl http://localhost:9200

# Expected response:
{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "8.11.0",
    ...
  }
}

# Check Kibana is running
curl http://localhost:5601/api/status

# Expected: HTTP 200 OK

First Look at Kibana

Once installed, open your browser to http://localhost:5601. You should see:

1. Welcome screen - Options to add sample data or explore on your own
2. Navigation menu - Left sidebar with various apps
3. Search bar - Global search across Kibana

Adding Sample Data

To get started quickly:

1. Click "Try sample data" on the home page
2. Choose one of the sample datasets:
   • Sample eCommerce orders - Retail analytics
   • Sample flight data - Travel and logistics
   • Sample web logs - Web server analysis
3. Click "Add data"
4. Click "View data" to see pre-built dashboard

This gives you data to explore as you learn!

Architecture Overview

Understanding how Kibana works under the hood:

┌─────────────────────────────────────┐
│         User Browser                │
│  (JavaScript, React Application)    │
└────────────┬────────────────────────┘
             │ HTTP/HTTPS
             │
┌────────────▼────────────────────────┐
│       Kibana Server                 │
│  - Node.js application              │
│  - Authentication & Authorization   │
│  - Query builder & aggregations     │
│  - Saved objects (dashboards, etc)  │
└────────────┬────────────────────────┘
             │ HTTP API
             │
┌────────────▼────────────────────────┐
│      Elasticsearch Cluster          │
│  - Stores indices                   │
│  - Executes queries                 │
│  - Returns results                  │
└─────────────────────────────────────┘

Key Points

• Kibana Server acts as a proxy between users and Elasticsearch
• All data lives in Elasticsearch, not Kibana
• Dashboards and visualizations are stored as "saved objects" in Elasticsearch
• User sessions and settings are also stored in Elasticsearch

System Requirements

Minimum Requirements

• RAM: 2GB (4GB recommended)
• CPU: 2 cores
• Disk: 10GB free space
• Browser: Chrome, Firefox, Safari, or Edge (latest versions)

Production Requirements

• RAM: 8GB+ (depends on data volume)
• CPU: 4+ cores
• Disk: SSD recommended, 50GB+ for logs
• Network: Low latency connection to Elasticsearch

Configuration Basics

Kibana's main configuration file is kibana.yml:

# Kibana port
server.port: 5601

# Elasticsearch connection
elasticsearch.hosts: ["http://localhost:9200"]

# Server host (0.0.0.0 for all interfaces)
server.host: "localhost"

# Enable security features
xpack.security.enabled: true

# Session timeout (in milliseconds)
xpack.security.session.idleTimeout: 3600000

Important Settings

Next Steps

Now that you understand what Kibana is and have it installed, you're ready to:

1. Explore the interface (Chapter 2)
2. Search your data (Chapter 3)
3. Create visualizations (Chapter 4)
4. Build dashboards (Chapter 5)

Common Installation Issues

Issue: Can't connect to Elasticsearch

# Check Elasticsearch is running
curl http://localhost:9200

# Check kibana.yml has correct elasticsearch.hosts
# Check firewall isn't blocking port 9200

Issue: Kibana won't start

# Check logs
tail -f /var/log/kibana/kibana.log

# Common causes:
# - Port 5601 already in use
# - Insufficient permissions
# - Incorrect Node.js version

Issue: Slow performance

# Increase Node.js memory
NODE_OPTIONS="--max-old-space-size=4096" kibana

# Adjust in kibana.yml
node.options: ["--max-old-space-size=4096"]

Summary

In this chapter, you learned:

• ✅ What Kibana is and its role in the Elastic Stack
• ✅ Key features and use cases
• ✅ How to install Kibana via multiple methods
• ✅ Basic architecture and configuration
• ✅ How to verify your installation

Next, we'll dive into the Kibana interface and start exploring data!