Building a Lightweight Local API Gateway + WAF with CrowdSec and Kafka

Modern APIs are constant targets for scanning, probing, and abuse. Attackers don’t need sophisticated exploits. Sometimes they just try common paths like /.env, /wp-admin, or /config.json and wait for something to leak.

In this post, I’ll walk through a simple but effective API gateway design that uses OpenResty as a reverse proxy, Kafka for request streaming, and CrowdSec for collaborative IP banning. The goal is not to build a heavyweight WAF, but a flexible pipeline where detection and enforcement are cleanly separated.

The High-Level Idea

1. OpenResty receives incoming HTTP requests and proxies them to backend services.
2. Each request is published to a Kafka topic for asynchronous inspection.
3. A separate WAF service consumes the Kafka stream, detects malicious behavior, and bans IPs using CrowdSec.

Why OpenResty as the API Gateway

OpenResty is built on NGINX but adds Lua scripting, which makes it ideal for this use case.

With OpenResty, you can:

• Act as a high-performance reverse proxy
• Inspect requests without blocking them
• Extract metadata like IP, path, headers, and method
• Publish messages to Kafka using Lua libraries or HTTP bridges

The key point is that OpenResty stays fast. It does not wait for security decisions. It just forwards requests and emits events.

Publishing Requests to Kafka

Every incoming HTTP request that passes through OpenResty is converted into a small event and sent to a Kafka topic. This event does not contain the full request body. It only includes the information needed for security analysis. Typical fields might include:

• Client IP address
• HTTP method
• Request path
• User-Agent
• Timestamp

For example, a request like:

GET https://www.example.com/.env

GET / .env HTTP/1.1
Host: www.example.com
User-Agent: curl/7.81.0
X-Forwarded-For: 185.220.101.42

After OpenResty receives the request, it publishes a message like this:

{
  "ip": "185.220.101.42",
  "method": "GET",
  "path": "/.env",
  "user_agent": "curl/7.81.0",
  "timestamp": "2026-02-06T10:15:42Z"
}

Kafka is a good fit here because:

• It handles high throughput easily
• Producers (OpenResty) and consumers (WAF) are decoupled
• You can replay traffic for testing or tuning detection rules

The WAF (Web Application Firewall) as a Kafka Consumer

Instead of inspecting traffic inline, the WAF runs as a Kafka consumer.

Its job is simple:

• Read request events from Kafka
• Apply detection logic
• Decide whether an IP is malicious

Detection can start very basic:

import java.util.Set;

public class SimpleDetector {

    private static final Set BLOCKED_PATHS = Set.of(
        "/.env",
        "/.git",
        "/config"
    );

    private static final Set BLOCKED_PREFIXES = Set.of(
        "/wp-admin",
        "/phpmyadmin"
    );

    public static boolean isMalicious(String path) {
        if (path == null || path.isBlank()) {
            return false;
        }

        String p = path.toLowerCase();

        if (BLOCKED_PATHS.contains(p)) {
            return true;
        }

        if (BLOCKED_PREFIXES.stream().anyMatch(p::startsWith)) {
            return true;
        }

        return p.contains("../") || p.contains("..%2f");
    }
}

Banning IPs with CrowdSec

When the WAF detects malicious behavior, it sends a decision to CrowdSec.

CrowdSec is responsible for:

• Managing IP reputation
• Applying ban durations
• Sharing threat signals with the community
• Distributing decisions to enforcement bouncers (firewall, NGINX/OpenResty, etc.)

Once an IP is banned, enforcement depends on the bouncer in use. With a firewall-level bouncer, traffic from the banned IP is blocked at the network layer and never reaches OpenResty again. Only the initial request that triggered detection is seen by the gateway.

Optionally, an OpenResty or NGINX bouncer can be used as an additional safety net to reject requests at the application layer.

This approach turns a single malicious request into lasting protection while keeping request handling fast and lightweight.

Final Thoughts

This setup exists for one simple reason: keep the API fast while still stopping obvious attacks.

OpenResty focuses on what it does best, handling requests and proxying traffic with minimal overhead. Security analysis is pushed out of the request path and handled asynchronously through Kafka, so normal users never pay the cost of detection.

Kafka gives flexibility. You can inspect traffic, improve rules, replay real requests, and scale detection without touching the gateway.

CrowdSec turns detection into action. Once an IP shows malicious behavior, it’s blocked early, often at the firewall, and never reaches your API again.

This approach works well for real-world APIs where most threats are automated scans, not complex exploits. It’s simple, fast, and easy to evolve as your needs grow.