Application Layer Protocols
Learning Objectives
By the end of this reading, you will be able to:
- Understand the role of application layer protocols
- Explain how HTTP and HTTPS work
- Describe the DNS resolution process
- Compare email protocols (SMTP, POP3, IMAP)
- Understand FTP and modern alternatives
- Work with REST APIs and HTTP methods
- Analyze real-world application layer communication
Introduction
The application layer is where network-aware software operates. This is the layer users interact with directly through web browsers, email clients, file transfer programs, and countless other applications.
Application layer protocols define how applications exchange data over a network. Unlike lower layers that focus on getting data from point A to point B, the application layer focuses on the meaning and structure of that data.
HTTP (Hypertext Transfer Protocol)
HTTP is the foundation of the World Wide Web, used for transferring web pages, images, videos, and other resources.
HTTP Characteristics
- Request-Response Protocol: Client sends request, server sends response
- Stateless: Each request is independent (no memory of previous requests)
- Text-Based: Human-readable format
- TCP-Based: Uses TCP port 80
- Flexible: Can transfer any type of data
HTTP Request Structure
GET /index.html HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
Accept: text/html,application/xhtml+xml
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
[Optional Body]
Components:
- Request Line: Method, URI, HTTP version
- Headers: Metadata about the request
- Blank Line: Separates headers from body
- Body: Optional data (used in POST, PUT)
HTTP Methods
| Method | Purpose | Idempotent | Safe |
|---|---|---|---|
| GET | Retrieve resource | Yes | Yes |
| POST | Submit data, create resource | No | No |
| PUT | Update/replace resource | Yes | No |
| PATCH | Partially update resource | No | No |
| DELETE | Delete resource | Yes | No |
| HEAD | Get headers only (no body) | Yes | Yes |
| OPTIONS | Get supported methods | Yes | Yes |
| TRACE | Echo request (debugging) | Yes | Yes |
Idempotent: Multiple identical requests have the same effect as one request Safe: Does not modify resources
HTTP Response Structure
HTTP/1.1 200 OK
Date: Mon, 23 Dec 2024 10:00:00 GMT
Server: Apache/2.4.41
Content-Type: text/html; charset=UTF-8
Content-Length: 1234
Last-Modified: Mon, 23 Dec 2024 09:00:00 GMT
ETag: "3f80f-1b6-5d9a3e00"
Connection: keep-alive
<!DOCTYPE html>
<html>
<head><title>Example</title></head>
<body><h1>Hello, World!</h1></body>
</html>
Components:
- Status Line: HTTP version, status code, status message
- Headers: Metadata about the response
- Blank Line: Separates headers from body
- Body: The actual content (HTML, JSON, image data, etc.)
HTTP Status Codes
1xx: Informational
100 Continue
101 Switching Protocols
2xx: Success
200 OK - Request succeeded
201 Created - Resource created
204 No Content - Success, but no body
3xx: Redirection
301 Moved Permanently - Resource permanently moved
302 Found - Resource temporarily moved
304 Not Modified - Cached version is current
4xx: Client Error
400 Bad Request - Invalid syntax
401 Unauthorized - Authentication required
403 Forbidden - Permission denied
404 Not Found - Resource doesn't exist
405 Method Not Allowed - Method not supported
429 Too Many Requests - Rate limit exceeded
5xx: Server Error
500 Internal Server Error - Generic server error
502 Bad Gateway - Invalid upstream response
503 Service Unavailable - Server overloaded/down
504 Gateway Timeout - Upstream timeout
HTTP Headers
Request Headers:
Host: www.example.com (Required in HTTP/1.1)
User-Agent: Mozilla/5.0... (Browser/client identification)
Accept: text/html (Acceptable content types)
Accept-Language: en-US (Preferred languages)
Accept-Encoding: gzip, deflate (Supported compression)
Cookie: session=abc123 (Session/tracking data)
Referer: https://google.com (Previous page)
Authorization: Bearer token123 (Authentication credentials)
Response Headers:
Content-Type: text/html (MIME type of content)
Content-Length: 1234 (Size in bytes)
Content-Encoding: gzip (Compression applied)
Set-Cookie: session=abc123 (Set cookie on client)
Cache-Control: max-age=3600 (Caching instructions)
Location: https://example.com/new (Redirect target)
Server: Apache/2.4.41 (Server software)
ETag: "3f80f-1b6" (Resource version identifier)
HTTP/1.1 vs HTTP/2 vs HTTP/3
HTTP/1.1 (1997):
- Text-based
- One request per TCP connection (or sequential with keep-alive)
- Head-of-line blocking
HTTP/2 (2015):
- Binary protocol
- Multiplexing: Multiple requests over single TCP connection
- Server push
- Header compression (HPACK)
HTTP/3 (2022):
- Based on QUIC (UDP-based)
- Faster connection establishment
- Better performance on lossy networks
- No head-of-line blocking at transport layer
Example HTTP Exchange
Client Server (www.example.com)
| |
| TCP Connection (3-way handshake) |
|<==========================================> |
| |
| GET /api/users/123 HTTP/1.1 |
| Host: www.example.com |
| Accept: application/json |
|---------------------------------------------->|
| |
| HTTP/1.1 200 OK |
| Content-Type: application/json |
| {"id": 123, "name": "John"} |
|<----------------------------------------------|
| |
| Connection closed or kept alive |
HTTPS (HTTP Secure)
HTTPS is HTTP over TLS/SSL, providing encryption, authentication, and integrity.
HTTPS Characteristics
- Encrypted: Data encrypted in transit
- Authenticated: Server identity verified (client optionally verified)
- Integrity: Data cannot be modified without detection
- Port: TCP 443
- Certificate-based: Uses X.509 certificates
HTTPS Connection Process
1. TCP Handshake (3 steps)
2. TLS Handshake (4 steps):
a) Client Hello
- Supported TLS versions
- Supported cipher suites
- Random number
b) Server Hello
- Selected TLS version
- Selected cipher suite
- Server certificate (contains public key)
- Random number
c) Client Key Exchange
- Verifies certificate
- Generates pre-master secret
- Encrypts with server's public key
- Sends encrypted pre-master secret
d) Session Keys Generated
- Both sides generate session keys from:
* Client random
* Server random
* Pre-master secret
- "Finished" messages exchanged (encrypted)
3. Encrypted HTTP communication begins
Certificate Chain
┌─────────────────────────┐
│ Root CA Certificate │ (Trusted by browser/OS)
│ (e.g., DigiCert) │
└────────────┬────────────┘
│ signs
▼
┌─────────────────────────┐
│ Intermediate Certificate │
└────────────┬────────────┘
│ signs
▼
┌─────────────────────────┐
│ Server Certificate │
│ (www.example.com) │
└─────────────────────────┘
Certificate contains:
- Domain name
- Public key
- Issuer (CA)
- Validity period
- Digital signature
DNS (Domain Name System)
DNS translates human-readable domain names to IP addresses.
DNS Hierarchy
. (Root)
|
┌───────────┼───────────┐
| | |
com org net (TLDs - Top Level Domains)
| | |
┌───┴───┐ ┌───┴───┐ ┌───┴───┐
example google wikipedia
| | |
www mail www (Subdomains)
DNS Record Types
| Type | Purpose | Example |
|---|---|---|
| A | IPv4 address | example.com → 93.184.216.34 |
| AAAA | IPv6 address | example.com → 2606:2800:220:1... |
| CNAME | Canonical name (alias) | www → example.com |
| MX | Mail server | example.com → mail.example.com |
| NS | Name server | example.com → ns1.example.com |
| TXT | Text (verification, SPF, etc.) | Various purposes |
| SOA | Start of Authority | Zone information |
| PTR | Reverse lookup (IP to name) | 34.216.184.93 → example.com |
DNS Resolution Process
User types: www.example.com
1. Browser cache
└─> Not found
2. OS cache
└─> Not found
3. Router cache (if configured)
└─> Not found
4. ISP DNS Resolver (Recursive Resolver)
└─> Not in cache, begins recursive query:
a) Query Root DNS Server
"Where is .com?"
→ Returns: "Ask TLD servers for .com"
b) Query .com TLD Server
"Where is example.com?"
→ Returns: "Ask ns1.example.com"
c) Query Authoritative Name Server (ns1.example.com)
"What is the IP for www.example.com?"
→ Returns: "93.184.216.34"
d) ISP Resolver caches result (TTL: time-to-live)
Returns IP to client
5. Browser connects to 93.184.216.34
DNS Query Types
Recursive Query:
- Client asks DNS server to fully resolve the name
- Server does all the work
Iterative Query:
- Server responds with best information it has
- May be a referral to another server
Non-Recursive Query:
- Answer already in cache
DNS Message Format
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| ID |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|QR| Opcode |AA|TC|RD|RA| Z | RCODE |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| QDCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| ANCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| NSCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| ARCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| Question |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| Answer |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| Authority |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| Additional |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
DNS Security: DNSSEC
DNSSEC adds cryptographic signatures to DNS records to prevent spoofing.
Email Protocols
SMTP (Simple Mail Transfer Protocol)
SMTP is used for sending email.
Characteristics:
- Port: 25 (unencrypted), 587 (STARTTLS), 465 (SSL/TLS)
- Text-based: Human-readable commands
- Push protocol: Sender pushes to receiver
SMTP Communication:
Client Server
| |
| TCP Connection to port 25/587 |
|-------------------------------------->|
| |
| 220 mail.example.com ESMTP |
|<--------------------------------------|
| |
| HELO client.domain.com |
|-------------------------------------->|
| |
| 250 Hello client.domain.com |
|<--------------------------------------|
| |
| MAIL FROM:<sender@example.com> |
|-------------------------------------->|
| |
| 250 OK |
|<--------------------------------------|
| |
| RCPT TO:<recipient@example.com> |
|-------------------------------------->|
| |
| 250 OK |
|<--------------------------------------|
| |
| DATA |
|-------------------------------------->|
| |
| 354 Start mail input |
|<--------------------------------------|
| |
| From: sender@example.com |
| To: recipient@example.com |
| Subject: Test Email |
| |
| This is the email body. |
| . |
|-------------------------------------->|
| |
| 250 OK: Message accepted |
|<--------------------------------------|
| |
| QUIT |
|-------------------------------------->|
| |
| 221 Bye |
|<--------------------------------------|
POP3 (Post Office Protocol v3)
POP3 is used for retrieving email.
Characteristics:
- Port: 110 (unencrypted), 995 (SSL/TLS)
- Download and Delete: Typically removes from server
- Simple: Minimal server-side operations
- Single Device: Not ideal for multiple devices
POP3 Session:
Client Server
| |
| TCP Connection to port 110 |
|---------------------------------->|
| |
| +OK POP3 server ready |
|<----------------------------------|
| |
| USER john@example.com |
|---------------------------------->|
| |
| +OK Password required |
|<----------------------------------|
| |
| PASS secretpassword |
|---------------------------------->|
| |
| +OK Logged in |
|<----------------------------------|
| |
| STAT |
|---------------------------------->|
| |
| +OK 3 messages (1200 bytes) |
|<----------------------------------|
| |
| RETR 1 |
|---------------------------------->|
| |
| [Message 1 content] |
|<----------------------------------|
| |
| DELE 1 |
|---------------------------------->|
| |
| +OK Message deleted |
|<----------------------------------|
| |
| QUIT |
|---------------------------------->|
| |
| +OK Signing off |
|<----------------------------------|
IMAP (Internet Message Access Protocol)
IMAP is a more advanced email retrieval protocol.
Characteristics:
- Port: 143 (unencrypted), 993 (SSL/TLS)
- Server Storage: Messages stay on server
- Folder Management: Create, delete, rename folders
- Selective Download: Download parts of messages
- Multiple Devices: Synchronized across devices
- Search: Server-side search capabilities
IMAP Features:
- Message flags (Read, Deleted, Flagged, etc.)
- Folder hierarchies
- Partial message fetching (headers, body parts)
- Server-side search
- Multiple mailbox support
- Message state synchronization
Email Protocol Comparison
| Feature | POP3 | IMAP |
|---|---|---|
| Message Storage | Client (usually) | Server |
| Multiple Devices | Poor | Excellent |
| Offline Access | Good | Limited to cached |
| Server Storage | Minimal | All messages |
| Folder Management | No | Yes |
| Partial Download | No | Yes |
| Server Search | No | Yes |
| Complexity | Simple | Complex |
FTP (File Transfer Protocol)
FTP transfers files between client and server.
FTP Characteristics
- Two Connections:
- Control connection (port 21): Commands
- Data connection (port 20): File transfers
- Authentication: Username/password
- Modes: Active and Passive
- Binary/ASCII: Transfer modes
FTP Session
Client Server
| |
| Control Connection (port 21) |
|-------------------------------------->|
| |
| 220 FTP server ready |
|<--------------------------------------|
| |
| USER username |
|-------------------------------------->|
| |
| 331 Password required |
|<--------------------------------------|
| |
| PASS password |
|-------------------------------------->|
| |
| 230 User logged in |
|<--------------------------------------|
| |
| PWD |
|-------------------------------------->|
| |
| 257 "/" is current directory |
|<--------------------------------------|
| |
| PASV (Passive mode) |
|-------------------------------------->|
| |
| 227 Entering Passive Mode (ip,port)|
|<--------------------------------------|
| |
| RETR filename.txt |
|-------------------------------------->|
| |
| [Data connection established] |
|<=====================================>|
| [File data transferred] |
|<=====================================>|
| |
| 226 Transfer complete |
|<--------------------------------------|
| |
| QUIT |
|-------------------------------------->|
| |
| 221 Goodbye |
|<--------------------------------------|
Active vs Passive FTP
Active FTP:
Client Server
| Control: Client → Server (port 21) |
| Data: Server → Client (client port) |
Problem: Firewalls block incoming connections to client
Passive FTP:
Client Server
| Control: Client → Server (port 21) |
| Data: Client → Server (random port) |
Solution: Client initiates both connections (firewall-friendly)
Modern Alternatives to FTP
SFTP (SSH File Transfer Protocol):
- Uses SSH (port 22)
- Encrypted
- Single connection
FTPS (FTP Secure):
- FTP over SSL/TLS
- Encrypted
- More complex than SFTP
SCP (Secure Copy):
- Based on SSH
- Simple file copying
- No directory listing/management
HTTP/HTTPS:
- File uploads via POST/PUT
- Downloads via GET
- REST APIs for file management
REST APIs
REST (Representational State Transfer) is an architectural style for web services.
REST Principles
- Client-Server: Separation of concerns
- Stateless: Each request contains all needed information
- Cacheable: Responses must define if cacheable
- Uniform Interface: Standardized communication
- Layered System: Client doesn't know if connected to end server
- Code on Demand (optional): Server can send executable code
RESTful API Design
Resource-Based URLs:
Good:
GET /api/users (Get all users)
GET /api/users/123 (Get user 123)
POST /api/users (Create user)
PUT /api/users/123 (Update user 123)
PATCH /api/users/123 (Partially update user 123)
DELETE /api/users/123 (Delete user 123)
Bad:
GET /api/getUser?id=123
POST /api/createUser
POST /api/updateUser
POST /api/deleteUser
HTTP Methods in REST
| Method | CRUD | Idempotent | Safe | Use Case |
|---|---|---|---|---|
| GET | Read | Yes | Yes | Retrieve resource(s) |
| POST | Create | No | No | Create new resource |
| PUT | Update | Yes | No | Replace entire resource |
| PATCH | Update | No | No | Partially update resource |
| DELETE | Delete | Yes | No | Remove resource |
REST API Example
Request:
POST /api/users HTTP/1.1
Host: api.example.com
Content-Type: application/json
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
{
"name": "John Doe",
"email": "john@example.com",
"role": "user"
}
Response:
HTTP/1.1 201 Created
Content-Type: application/json
Location: /api/users/456
{
"id": 456,
"name": "John Doe",
"email": "john@example.com",
"role": "user",
"created_at": "2024-12-23T10:00:00Z"
}
API Authentication
1. API Keys:
GET /api/data HTTP/1.1
X-API-Key: abc123def456ghi789
2. Bearer Token (JWT):
GET /api/data HTTP/1.1
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
3. OAuth 2.0:
User → Application → Authorization Server → Resource Server
4. Basic Auth:
GET /api/data HTTP/1.1
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
JSON (JavaScript Object Notation)
The most common data format for REST APIs:
{
"users": [
{
"id": 1,
"name": "Alice",
"email": "alice@example.com",
"roles": ["admin", "user"],
"active": true,
"profile": {
"age": 30,
"city": "New York"
}
},
{
"id": 2,
"name": "Bob",
"email": "bob@example.com",
"roles": ["user"],
"active": false,
"profile": null
}
],
"total": 2,
"page": 1
}
REST API Status Codes
Success:
200 OK - Standard success
201 Created - Resource created
204 No Content - Success with no body
Client Errors:
400 Bad Request - Invalid syntax/data
401 Unauthorized - Authentication required
403 Forbidden - Authenticated but not authorized
404 Not Found - Resource doesn't exist
409 Conflict - Conflict with current state
422 Unprocessable - Validation failed
429 Too Many Requests - Rate limit exceeded
Server Errors:
500 Internal Server Error
502 Bad Gateway
503 Service Unavailable
WebSockets
WebSockets provide full-duplex communication over a single TCP connection.
WebSocket vs HTTP
HTTP:
Client → Server: Request
Server → Client: Response
[Connection may close]
WebSocket:
Client → Server: Upgrade request
Server → Client: Upgrade acknowledgment
Client ↔ Server: Bidirectional messages
[Persistent connection]
WebSocket Handshake
Client Request:
GET /chat HTTP/1.1
Host: example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
Server Response:
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
[WebSocket connection established]
Use Cases:
- Real-time chat
- Live notifications
- Multiplayer games
- Stock tickers
- Collaborative editing
Exercises
Basic Exercises
HTTP Status Codes: What status code would you return for each scenario?
- a) User successfully logs in
- b) User tries to access admin page without permission
- c) Requested resource doesn't exist
- d) Server database is down
- e) Successfully deleted a user
HTTP Methods: Which HTTP method would you use for:
- a) Retrieving a list of products
- b) Creating a new blog post
- c) Updating a user's email address
- d) Deleting a comment
- e) Replacing an entire product record
DNS Resolution: List the steps in order when resolving "www.github.com":
- a) What servers are queried?
- b) What type of DNS record is returned?
- c) Where might the answer be cached?
Email Protocols: Which protocol (SMTP, POP3, IMAP) would you use for:
- a) Sending an email
- b) Checking email on multiple devices
- c) Downloading email to a single laptop with limited server storage
Intermediate Exercises
REST API Design: Design RESTful endpoints for a blog with posts and comments:
- a) Get all posts
- b) Get a specific post
- c) Create a new post
- d) Update a post
- e) Delete a post
- f) Get all comments for a post
- g) Add a comment to a post
- h) Delete a comment
HTTP Headers: Explain the purpose of these headers:
- a)
Content-Type: application/json - b)
Authorization: Bearer token123 - c)
Cache-Control: no-cache - d)
Accept-Encoding: gzip, deflate - e)
Set-Cookie: session=abc; HttpOnly; Secure
- a)
HTTPS Certificate: Explain what happens if:
- a) Certificate is expired
- b) Domain name doesn't match certificate
- c) Certificate is self-signed
- d) Certificate chain is broken
API Response Design: Design JSON responses for:
- a) Successful user creation
- b) Validation error (missing required fields)
- c) Paginated list of 100 users (10 per page)
- d) Resource not found error
Advanced Exercises
Complete API Implementation: Design a complete REST API for a library system:
- Resources: Books, Authors, Users, Loans
- Include: Endpoints, methods, request/response formats, status codes
- Relationships: Books have authors, users borrow books
- Authentication: Who can perform which operations?
DNS Analysis: Trace the complete DNS resolution for "mail.google.com":
- a) Show each query (recursive and iterative)
- b) Identify which DNS servers are contacted
- c) What type of record is mail.google.com?
- d) How long might this result be cached?
HTTP vs HTTPS Performance: Compare loading a web page over HTTP vs HTTPS:
- a) How many round trips for HTTP?
- b) How many round trips for HTTPS (including TLS handshake)?
- c) What are the performance trade-offs?
- d) When would HTTP/2 improve performance?
Email Flow: Trace the complete path of an email from alice@example.com to bob@different.com:
- a) What protocols are used?
- b) What DNS queries occur?
- c) How many servers are involved?
- d) What happens if the recipient server is down?
API Rate Limiting: Design a rate limiting strategy for an API:
- Different limits for: Anonymous users, authenticated users, premium users
- Response format when limit exceeded
- Headers to communicate rate limit status
- How to handle burst traffic
Summary
In this reading, we explored the major application layer protocols:
- HTTP/HTTPS: The foundation of the web, with HTTPS adding security through TLS/SSL
- DNS: Translates domain names to IP addresses through a hierarchical system
- Email Protocols:
- SMTP for sending
- POP3 for simple retrieval
- IMAP for advanced, multi-device email access
- FTP: Traditional file transfer (largely replaced by SFTP/HTTPS)
- REST APIs: Modern web services using HTTP methods and JSON
- WebSockets: Full-duplex, persistent connections for real-time communication
These protocols enable the applications we use daily: web browsing, email, file sharing, and API-driven services.
Key Takeaways
- HTTP is stateless and request-response based; HTTPS adds encryption
- Status codes communicate the outcome of HTTP requests (2xx success, 4xx client error, 5xx server error)
- DNS uses a hierarchical system with caching at multiple levels
- IMAP is generally preferred over POP3 for modern email access
- REST APIs use HTTP methods (GET, POST, PUT, DELETE) to operate on resources
- JSON has become the de facto standard for API data exchange
- Authentication is critical for API security (API keys, tokens, OAuth)
Next Steps
Now that you understand application layer protocols, you're ready to explore network security. In the next reading, we'll cover:
- Firewalls and network access control
- Encryption fundamentals
- TLS/SSL in depth
- VPNs and secure tunneling
- Common network attacks and defenses
Continue to: 04-network-security.md
Additional Resources
- RFC 2616: HTTP/1.1
- RFC 7540: HTTP/2
- RFC 9114: HTTP/3
- RFC 1035: Domain Names
- RFC 5321: SMTP
- RFC 1939: POP3
- RFC 3501: IMAP
- RFC 959: FTP
- MDN Web Docs: HTTP
- REST API Tutorial
This reading is part of Module 8: Networking