I've built and maintained my fair share of webhook gateways and integration layers. A few common pain points kept cropping up:

1. The Code Bottleneck: Every time a new webhook source or a slight variation in transformation logic was needed, it meant code changes. A new handler, a modified Pydantic model, a redeploy. I, or my team, became the bottleneck.
2. Ownership Tangles: Giving multiple teams or users the ability to define their own webhook transformations often meant granting them broader application deployment privileges, or setting up complex, isolated microservices for each. Neither felt quite right.
3. Repetitive Logic: So many webhook handlers do similar things: pick a few fields, rename some keys, maybe enrich with a static lookup. Writing Python for output_payload['userName'] = input_payload['user']['login'] over and over felt like I was missing a more declarative way.
4. Observability Challenges: Understanding why a specific webhook failed or was transformed in a certain way often involved digging through application logs, which could be scattered or inconsistently formatted.

These issues led me to wonder: could there be a more data-centric, self-service approach?

This is where my mental model for DuckDB started to shift.

Think about it:

• We trust SQL to define data contracts and schemas (hello, dbt users!).
• We trust SQL to express complex aggregations, filters, and joins for analytics.
• SQL is declarative, its behavior is (usually) inspectable, and its syntax is widely understood.

So, why not trust SQL to express the live transformation and routing logic for something like webhooks? The key insight for me was this: if the logic is defined as data (SQL queries stored in a table), and the engine to execute that logic is also data-aware (DuckDB), then you unlock a powerful new paradigm.

In the system I've built, the database isn't just a passive recipient of data; it's the active processing pipeline.

The database isn't just a data sink; it's the pipeline itself. It’s almost like turning your middleware inside out.

This might seem like an academic exercise, but it unlocks some genuinely compelling properties.

✅ Declarative Self-Service: Teams can define and update their webhook transformations and filters by submitting SQL queries via an API (or the provided UI!). A full set of API endpoints allows managing webhooks (register, update, list, activate/deactivate, delete), reference tables, and UDFs.

No application redeploys are needed for logic changes. They own their logic.

✅ Queryable Runtime State: Every raw event, every transformation attempt, its success or failure, and the final payload are logged within DuckDB.

Need to debug why a webhook for /github-events isn't working? The /query endpoint or direct DB access lets you run SQL like:

This level of direct, SQL-based introspection into the runtime behavior is incredibly powerful. Specific API endpoints also exist to view recent events and drill down into individual event details.

✅ Composable & Extensible Logic:

Need to enrich webhook data with user details?

Upload a users.csv or JSON file as a reference table (/upload_table) and JOIN it in your transform query using its namespaced name (e.g., ref_webhook123_users).

Need a complex string manipulation or a call to an external validation service that SQL can't easily do?

Register a Python UDF (/register_udf) and call it directly from your SQL using its namespaced name (e.g., udf_webhook123_extract_jira_key).

✅ Portable Artifact & Simplified Operations: The entire state of the gateway - configurations, reference data, UDF definitions, and logs - can live inside a single DuckDB file (though the path is configurable via the DUCKDB_PATH environment variable). Backups are simple file copies.

✅ "Infra as Data": The behavior of the system is defined by data (SQL queries, reference tables) stored within the database, rather than by imperative code deployed in an application layer.

✅ Robust Implementation: Under the hood, it uses standard Python async capabilities (asyncio, FastAPI) along with thread pools and locking (ThreadPoolExecutor, asyncio.Lock) to handle requests concurrently while ensuring safe access to the DuckDB database. Database operations are wrapped in transactions for atomicity.

And those are valid points if your primary goal is just to write a simple webhook handler quickly. But this approach isn't trying to optimize for the simplest possible "hello world" webhook. It's aiming to solve a deeper set of problems around making runtime behavior highly configurable via data, and shifting who owns the transformation logic. It's about enabling a more federated, self-service model.

Imagine we want to process GitHub push events for the main branch only. Here's how it works:

This simple flow, driven entirely by SQL configurations stored in the database, demonstrates the core power.

It's primarily an exploration. But it’s an exciting one because it forces us to invert the typical application stack.

I see this pattern as a prototype for thinking differently about:

• Data-driven control planes: Where system behavior is dynamically configured through data.
• Self-service data pipelines: Empowering users to build their own simple ETLs with SQL, reference data, and custom Python logic.
• Extensible audit layers: Creating rich, queryable logs of system activity by default.
• Dynamic caching layers: Imagine SQL defining cache invalidation or transformation logic.
• Per-tenant customization: Using SQL to define tenant-specific routing or data shaping.

It’s not just about this specific webhook gateway. It’s about advocating for a way of thinking: that databases, especially modern ones like DuckDB, can be potent runtime logic engines, not just passive storage layers. And that sometimes, "weird" architectures unlock remarkably useful properties once you reframe the problem you're trying to solve.

I'm curious to hear what others think. What are the pitfalls I haven't considered? What other use cases could this "database as runtime" pattern unlock? Let me know!