Before Lab 4.3 — Application Telemetry and Correlation IDs

Your application is the best source of security truth. Platform logs show that a request arrived; your application logs show what happened when the request was processed.

---

Application Insights: where telemetry goes

Application Insights is Azure's Application Performance Management (APM) service. It collects telemetry from your application — requests, dependencies, exceptions, and custom events — and stores it in Log Analytics where you can query it with KQL.

Application Insights collects four main types of telemetry:

Type	Captures	Example KQL table
Requests	HTTP requests, status codes, duration	AppRequests
Dependencies	Calls to databases, HTTP endpoints, Azure SDK calls	AppDependencies
Exceptions	Unhandled errors with stack traces	AppExceptions
Custom events	Business-level events you instrument yourself	AppTraces

All telemetry includes a correlation ID (from the x-correlation-id header) and an operation_Id (generated automatically by the SDK). The correlation ID lets you trace a specific client request across services; the operation_Id links all records from that single request together.

---

Why application-level telemetry matters

Platform logs (from Diagnostic Settings) show the HTTP layer. Application-level telemetry shows what your code did with that request. This distinction is critical for security investigations.

What platform logs tell you: A request from IP 203.0.113.5 reached the API and received a 200 response.

What application logs tell you: The request was from an authenticated user, who read a secret from Key Vault, with correlation ID 01111111-1111-4111-b111-111111111113.

For security investigations, you need both. The platform log gives you the network context; the application log gives you the business context.

---

Structured logging

Structured logging means logging data in a machine-readable format (JSON) rather than free-form text:

// Structured log entry
{
  "timestamp": "2024-01-15T10:30:00.123Z",
  "service": "cloud-security-demo",
  "event": "secret-accessed",
  "correlationId": "abc-123",
  "user": "app-service-managed-identity",
  "key_vault_secret": "demo-api-message",
  "source_ip": "203.0.113.5"
}

Structured logs are queryable by Log Analytics. Free-form logs like "Accessed secret demo-api-message for user app-service-managed-identity" are much harder to search reliably.

Benefits of structured logging

Structured	Free-form
Queryable by field	Requires regex matching
Consistent format	Inconsistent between developers
Includes metadata (timestamps, services)	Missing context
Machine-readable (alerts, dashboards)	Human-readable only

---

Correlation IDs: tracing across services

A correlation ID is a unique identifier that is passed through every layer of a request's journey. It enables you to trace a single request across multiple services.

How correlation IDs work

Step 1: Client generates a unique correlation ID (UUID) and sends it in the x-correlation-id HTTP header.

Step 2: Public backend reads the header, adds it to all log entries, and passes it to the internal backend.

Step 3: Internal backend reads the header, adds it to its log entries, and logs Key Vault access with the same ID.

Step 4: In Log Analytics, you query by correlation ID to see the entire request chain:

AppRequests
| where TimeGenerated > ago(1h)
| project Name, OperationId, TimeGenerated

Why correlation IDs matter for security

Consider this attack scenario:

1. Attacker sends request with correlation ID: attack-uuid-001
2. Public backend logs: "Received GET /api/data from 203.0.113.5, correlation ID: attack-uuid-001"
3. Internal backend logs: "Processed request, accessed Key Vault secret, correlation ID: attack-uuid-001"
4. Attacker sends second request with correlation ID: attack-uuid-002
5. Attacker sends third request with correlation ID: attack-uuid-002 (same ID)

With correlation IDs, you can:

• Group all actions by the same attacker request
• Identify attack chains where multiple operations use the same correlation ID
• Correlate platform logs with application logs using the same identifier
• Reconstruct the full attack from fragmented logs across services

Without correlation IDs, you would need to correlate by time and IP address alone — which is unreliable when multiple requests arrive simultaneously from the same IP.

---

OpenTelemetry-based SDK (@azure/monitor-opentelemetry)

The OpenTelemetry-based SDK automatically captures:

Auto-collected	Description
Requests	HTTP method, URL, status code, duration, response size
Dependencies	Calls to databases, HTTP calls, storage operations (including Azure SDK calls)
Exceptions	Stack traces and error messages

You can also add custom telemetry using the OpenTelemetry logs API:

// Import the logs API
import { logs, SeverityNumber } from "@opentelemetry/api-logs";

const logger = logs.getLogger("my-app");

logger.emit({
  body: "Secret accessed",
  severityNumber: SeverityNumber.INFO,
  attributes: {
    "microsoft.custom_event.name": "secret-accessed",
    correlationId,
    secretName: "demo-api-message",
    source: "app-service-managed-identity",
  },
});

Custom telemetry is critical for security monitoring because it captures business-level events that platform logs cannot see — such as which specific secret was accessed, by which user, and for what purpose.

---

Application code changes for Day 4

The application requires two key additions to enable proper telemetry:

Component	Purpose
Correlation ID middleware	Reads/generates x-correlation-id header and passes it through the request chain
OpenTelemetry-based SDK	Captures requests, dependencies, exceptions, and custom log events automatically

Correlation ID middleware

The correlation ID middleware reads the x-correlation-id header from incoming requests. If present, it uses that value; otherwise, it generates a UUID. The ID is stored on the request object for use in log entries and passed to downstream services:

// Middleware function (in apps/backend/src/correlation.ts)
function correlationMiddleware(req, res, next) {
  const incomingId = req.headers['x-correlation-id'];
  const correlationId = incomingId && isValidCorrelationId(incomingId)
    ? incomingId
    : crypto.randomUUID();

  req.correlationId = correlationId;
  res.setHeader('x-correlation-id', correlationId);
  next();
}

Structured logging

Each endpoint in the application uses a log() function that emits structured telemetry via the OpenTelemetry logs API:

// In apps/backend/src/logger.ts
import { logs, SeverityNumber } from "@opentelemetry/api-logs";

const logger = logs.getLogger(config.appName);

export function log(message, fields = {}) {
  logger.emit({
    body: `[${config.appName}] ${message}`,
    severityNumber: SeverityNumber.INFO,
    attributes: { ...fields, service: config.appName },
  });
}

OpenTelemetry-based SDK initialization

The SDK is initialized at the entry point of each backend (index.ts):

// Initialize SDK with connection string from environment
import { useAzureMonitor } from "@azure/monitor-opentelemetry";

useAzureMonitor({
  azureMonitorExporterOptions: {
    connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING || "",
  },
});

The SDK automatically captures HTTP requests, dependencies (including Azure SDK calls to Key Vault and Table Storage), exceptions, and performance data. Custom log events (such as "secret accessed") are emitted via the log() function described above.

---

Lab 4.3 objectives

Lab 4.3 goals

• Understand where correlation IDs are generated and how they flow through the request chain
• Identify the middleware components that enable structured logging
• Generate telemetry by accessing application endpoints with custom correlation IDs
• Trace a request from the public backend to the internal backend using correlation IDs
• Understand what data the OpenTelemetry-based SDK captures automatically vs. what you need to instrument

Key questions before starting

1. Why is a correlation ID more reliable than IP address + timestamp for request tracing?
2. What happens if a client does not send an x-correlation-id header?
3. Which is better for security monitoring: structured JSON logs or free-form text logs?
4. If the OpenTelemetry-based SDK is not installed, what security events will you miss?

---

Lab File

Download Lab 4.3 — Application Telemetry PDF

---

References

• Application Insights OpenTelemetry observability overview
• Enable OpenTelemetry in Application Insights
• Structured logging best practices