# flAPI — Instant REST APIs & MCP Tools from SQL on DuckDB

flAPI transforms SQL queries into production-ready REST APIs powered by DuckDB. Build blazing-fast, secure, AI-ready APIs in minutes.

# Integrating flAPI with Claude

Anthropic's Claude clients (Claude Desktop, Claude Code, the Anthropic SDK) speak the **Model Context Protocol** natively. flAPI exposes its tools, resources, and prompts over the JSON-RPC 2.0 HTTP transport at `POST /mcp/jsonrpc`, so any MCP-aware Claude client can connect to it once a thin transport bridge is in place.

This page covers the practical setup for the three most common Claude entry points.

## How flAPI's MCP Endpoint Works

*   One transport URL: `POST http://<host>:<port>/mcp/jsonrpc`
*   Optional session header echoed back by the server: `Mcp-Session-Id`
*   One liveness URL: `GET /mcp/health` (not part of MCP itself)
*   Authentication (when enabled) via `Authorization: Bearer <token>` or `Authorization: Basic ...`

If you have not already configured an `mcp-tool:` block on an endpoint, start with the [MCP Overview](/docs/ai-integration/mcp-overview.md).

## 1\. Claude Desktop

Claude Desktop's `mcpServers` configuration launches each MCP server as a **stdio subprocess**. Because flAPI speaks HTTP, you need a stdio→HTTP bridge. The recommended bridge is [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) (works for any HTTP-transport MCP server).

### Step 1 — Locate your config file

| OS  | Path |
| --- | --- |
| macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` |
| Windows | `%APPDATA%\Claude\claude_desktop_config.json` |
| Linux | `~/.config/Claude/claude_desktop_config.json` |

### Step 2 — Add the flAPI bridge

```
{  "mcpServers": {    "flapi": {      "command": "npx",      "args": [        "-y",        "mcp-remote",        "http://localhost:8080/mcp/jsonrpc"      ]    }  }}
```

If your flAPI server requires a Bearer token, pass an `Authorization` header through the bridge:

```
{  "mcpServers": {    "flapi": {      "command": "npx",      "args": [        "-y",        "mcp-remote",        "http://localhost:8080/mcp/jsonrpc",        "--header",        "Authorization: Bearer ${FLAPI_TOKEN}"      ],      "env": {        "FLAPI_TOKEN": "your-bearer-token"      }    }  }}
```

### Step 3 — Restart Claude Desktop

Quit Claude Desktop fully and reopen it. Your flAPI tools will appear in the "Tools" indicator and Claude can call them in conversation.

> **What about `"command": "curl"`?** Earlier examples in some places on the internet suggest using `curl` as the command. That does not work: Claude Desktop expects a long-lived stdio MCP process speaking JSON-RPC line frames, not a one-shot HTTP request. Use `mcp-remote` (or any equivalent stdio↔HTTP bridge such as `mcp-proxy`) until Claude Desktop ships native HTTP-transport support.

## 2\. Claude Code (CLI)

Claude Code supports MCP servers via the same `mcpServers` config block. The setup is identical to Claude Desktop—register the bridge and Claude Code will list flAPI tools alongside its built-ins:

```
{  "mcpServers": {    "flapi": {      "command": "npx",      "args": ["-y", "mcp-remote", "http://localhost:8080/mcp/jsonrpc"]    }  }}
```

If you are using flAPI's **Config Service** (`flapi --config-service`), Claude Code can also drive the `flapi_*` admin tools to create, update, or reload endpoints during a coding session. See the [MCP Config Tools](/docs/ai-integration/mcp-config-tools.md) page for the full tool catalog.

## 3\. Direct API Use (Anthropic SDK)

When you call Claude via the Anthropic API, you typically expose tools through the [Tool Use](https://docs.anthropic.com/claude/docs/tool-use) feature rather than through MCP transport. The easiest pattern is:

1.  Call flAPI's `tools/list` once at startup and translate each MCP tool definition into an Anthropic `tool` definition (the JSON Schema `inputSchema` maps directly to the Anthropic `input_schema` field).
2.  When Claude returns a `tool_use` block, call flAPI's `tools/call` with `name` and `arguments`.
3.  Feed the resulting `content` array back to Claude as a `tool_result`.

A minimal Python skeleton:

```
import requestsimport anthropicFLAPI = "http://localhost:8080/mcp/jsonrpc"client = anthropic.Anthropic()def jsonrpc(method, params=None, _id=1):    r = requests.post(FLAPI, json={        "jsonrpc": "2.0",        "id": _id,        "method": method,        "params": params or {},    })    return r.json()["result"]# 1) Initialise oncejsonrpc("initialize", {    "protocolVersion": "2025-11-25",    "clientInfo": {"name": "py-claude-bridge", "version": "1.0.0"},})# 2) Pull tools and translatemcp_tools = jsonrpc("tools/list")["tools"]anthropic_tools = [    {        "name": t["name"],        "description": t["description"],        "input_schema": t["inputSchema"],    }    for t in mcp_tools]# 3) Ask Clauderesponse = client.messages.create(    model="claude-opus-4-7",    max_tokens=1024,    tools=anthropic_tools,    messages=[{"role": "user", "content": "Top US campaigns by revenue?"}],)# 4) If Claude requests a tool, dispatch back to flAPIfor block in response.content:    if block.type == "tool_use":        result = jsonrpc("tools/call", {            "name": block.name,            "arguments": block.input,        })        # result["content"] is an array of content blocks ready        # to send back to Claude as a tool_result
```

This pattern works for any MCP server—the `inputSchema`→`input_schema` mapping is intentional in the MCP spec.

## Example Conversation

Once Claude is wired up, you can chat naturally:

```
You: "What are our top-performing marketing campaigns in the US?"Claude: Let me check that.[Tool call: get_campaign_performance { "country": "US" }]Claude: Based on the data, here are your top US campaigns this quarter:  1. Summer Sale 2024  — $45K revenue, 150K clicks (0.30 RPC)  2. Back to School    — $38K revenue, 120K clicks (0.32 RPC)  3. Mid-Year Refresh  — $31K revenue, 110K clicks (0.28 RPC)
```

Claude sees the full `inputSchema` you defined in YAML, so it knows which parameters are required, what shape they take, and what each one means. The richer your `description:` fields, the better its tool selection.

## Troubleshooting

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| No tools appear in Claude | Bridge cannot reach flAPI | Run `curl http://localhost:8080/mcp/health` |
| Tools listed but every call fails | Auth required by flAPI | Add `--header "Authorization: Bearer ..."` to `mcp-remote` |
| `Method not found` (-32601) | Method name typo or wrong protocol | Confirm method is one of `initialize`, `tools/list`, `tools/call`, `resources/list`, `resources/read`, `prompts/list`, `prompts/get`, `ping`, `logging/setLevel`, `completion/complete` |
| `Authentication required` (-32001) | flAPI's MCP auth is enabled and no/invalid token sent | Verify token; for per-method auth see the [Protocol Reference](/docs/ai-integration/mcp-protocol.md) |
| Stale tools after editing YAML | Endpoint not reloaded | Restart flAPI, or use `flapi_reload_endpoint` when running `--config-service` |

## More Information

*   **[MCP Overview](/docs/ai-integration/mcp-overview.md)** — concepts, configuration, security model
*   **[MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md)** — every JSON-RPC method, error code, and session header
*   **[MCP Config Tools](/docs/ai-integration/mcp-config-tools.md)** — admin tools for managing flAPI from Claude
*   **[YAML Syntax](/docs/endpoints/yaml-syntax.md)** — full `mcp-tool` / `mcp-resource` / `mcp-prompt` schema
*   **[Authentication](/docs/endpoints/authentication.md)** — securing the MCP transport

Need help wiring Claude into your data stack? [Contact our team](/services).

([🍪 Cookie Settings](#cookie-settings))

# MCP Config Tools (`flapi_*`)

In addition to the **data tools** you define yourself in YAML, flAPI ships a built-in family of **admin tools** that let an AI agent introspect and reconfigure the running flAPI server itself. These are exposed via the same MCP transport (`POST /mcp/jsonrpc`) and prefixed with `flapi_`.

They unlock workflows like:

*   "Show me the schema of the customer database, then create a `GET /customers` endpoint backed by `customers.sql`."
*   "Validate the Mustache template for the `orders` endpoint and hot-reload it."
*   "Refresh the cache for `dashboard_metrics` and show me the audit trail."

## Data Tools vs. Config Tools

|     | Data tools | Config tools |
| --- | --- | --- |
| Defined where | `mcp-tool:` blocks in your endpoint YAML | Built into the flAPI binary |
| Name pattern | Anything you choose (e.g. `get_orders`) | `flapi_*` |
| What they do | Execute SQL against your data | Manage endpoints, templates, caches |
| Always available? | Yes (whenever the endpoint is configured) | **Only with `--config-service`** |
| Authentication | Endpoint-level auth | Config service token (Bearer) for mutating tools |

Both kinds of tools appear in the same `tools/list` response and are invoked the same way via `tools/call`.

## Enabling Config Tools

The `flapi_*` tools are gated behind the **Config Service** feature, which must be enabled at startup:

```
# Enable config service (and config tools) with no auth./flapi --config-service# Recommended: also set an admin token./flapi --config-service --config-service-token "your-secret-token"
```

Without `--config-service`, flAPI's `ConfigToolAdapter` is not constructed, and any `flapi_*` call returns `-32603 Tool execution failed: Config tools not available`.

See also: [Configuration Service](/docs/tools/configuration-service.md) (REST counterpart and CLI).

## Authentication

The config tools share their authentication model with the REST Config Service:

*   **Read tools** (`flapi_get_*`, `flapi_list_*`, `flapi_expand_*`, `flapi_test_*`) are **unauthenticated** by default.
*   **Mutating tools** (`flapi_create_*`, `flapi_update_*`, `flapi_delete_*`, `flapi_reload_*`, `flapi_refresh_*`, `flapi_run_*`) require a valid token when one was set at startup with `--config-service-token`.

Pass the token as a Bearer header on each MCP request:

```
Authorization: Bearer your-secret-token
```

flAPI extracts the token in `MCPRouteHandlers::handleToolsCallRequest` and threads it to `ConfigToolAdapter::executeTool`.

## Tool Catalog

There are **19 `flapi_*` tools**, organised into four functional categories. (The upstream reference document is internally inconsistent—the header claims 20 and another section says 18—but counting the `### flapi_*` entries in `MCP_CONFIG_TOOLS_API.md` yields 19. Sources: `src/mcp_route_handlers.cpp`, `docs/MCP_CONFIG_TOOLS_API.md`.)

### Discovery (5 tools, read-only)

Introspect the project, environment, filesystem, and database schema.

| Tool | Args | Purpose |
| --- | --- | --- |
| `flapi_get_project_config` | —   | Project metadata: name, description, base path, version |
| `flapi_get_environment` | —   | Whitelisted environment variables visible to templates |
| `flapi_get_filesystem` | —   | Tree of YAML/SQL files under the project root |
| `flapi_get_schema` | —   | Aggregate schema (tables/columns) for all connections |
| `flapi_refresh_schema` | —   | Force a re-discovery of all connection schemas |

Example:

```
{  "jsonrpc": "2.0",  "id": 1,  "method": "tools/call",  "params": { "name": "flapi_get_schema", "arguments": {} }}
```

### Template (4 tools)

Read, write, expand, and validate SQL templates.

| Tool | Args | Auth | Purpose |
| --- | --- | --- | --- |
| `flapi_get_template` | `endpoint` | —   | Read the raw SQL template content |
| `flapi_update_template` | `endpoint`, `content` | Yes | Overwrite the template file on disk |
| `flapi_expand_template` | `endpoint`, `params?` | —   | Render Mustache with sample params (no SQL execution) |
| `flapi_test_template` | `endpoint` | —   | Validate Mustache/template syntax |

Example — preview how a template renders before saving:

```
{  "jsonrpc": "2.0",  "id": 2,  "method": "tools/call",  "params": {    "name": "flapi_expand_template",    "arguments": {      "endpoint": "customers",      "params": { "id": "42" }    }  }}
```

### Endpoint (6 tools)

CRUD plus hot-reload for REST/MCP endpoint configuration.

| Tool | Args | Auth | Purpose |
| --- | --- | --- | --- |
| `flapi_list_endpoints` | —   | —   | List all configured endpoints with type (`rest`/`mcp`) |
| `flapi_get_endpoint` | `path` | —   | Full configuration for one endpoint |
| `flapi_create_endpoint` | `path`, `method?`, `template_source?` | Yes | Create a new endpoint YAML |
| `flapi_update_endpoint` | `path`, `method?`, `template_source?` | Yes | Modify an existing endpoint |
| `flapi_delete_endpoint` | `path` | Yes | Remove an endpoint |
| `flapi_reload_endpoint` | `path` | Yes | Re-read the endpoint YAML from disk without restarting flAPI |

Example — let an agent create a new endpoint end-to-end:

```
{  "jsonrpc": "2.0",  "id": 3,  "method": "tools/call",  "params": {    "name": "flapi_create_endpoint",    "arguments": {      "path": "customers",      "method": "GET",      "template_source": "customers.sql"    }  }}
```

### Cache (4 tools)

Inspect and operate flAPI's DuckLake-backed endpoint cache.

| Tool | Args | Auth | Purpose |
| --- | --- | --- | --- |
| `flapi_get_cache_status` | `path` | —   | Is caching enabled? Which DuckLake table? |
| `flapi_refresh_cache` | `path` | Yes | Trigger a manual cache refresh for one endpoint |
| `flapi_get_cache_audit` | `path` | —   | Recent cache events (refreshes, status checks) |
| `flapi_run_cache_gc` | `path?` | Yes | Run garbage collection (per-endpoint or globally) |

Example — refresh a cache from an agent:

```
{  "jsonrpc": "2.0",  "id": 4,  "method": "tools/call",  "params": {    "name": "flapi_refresh_cache",    "arguments": { "path": "dashboard_metrics" }  }}
```

## Worked Example: "Add an endpoint by chat"

Here is the typical sequence an AI coding assistant runs when asked to create a new endpoint. All calls go to `POST /mcp/jsonrpc`; only the JSON body is shown.

```
// 1. Inspect the database{ "method": "tools/call",  "params": { "name": "flapi_get_schema", "arguments": {} } }// 2. See what already exists{ "method": "tools/call",  "params": { "name": "flapi_list_endpoints", "arguments": {} } }// 3. Create the endpoint{ "method": "tools/call",  "params": {    "name": "flapi_create_endpoint",    "arguments": {      "path": "customers",      "method": "GET",      "template_source": "customers.sql"    }  } }// 4. Write the SQL template{ "method": "tools/call",  "params": {    "name": "flapi_update_template",    "arguments": {      "endpoint": "customers",      "content": "SELECT id, name, email FROM customers WHERE 1=1 {{#params.id}}AND id = {{{params.id}}}{{/params.id}}"    }  } }// 5. Validate it{ "method": "tools/call",  "params": { "name": "flapi_test_template", "arguments": { "endpoint": "customers" } } }// 6. Hot-reload it into the running server{ "method": "tools/call",  "params": { "name": "flapi_reload_endpoint", "arguments": { "path": "customers" } } }// 7. Verify it{ "method": "tools/call",  "params": { "name": "flapi_get_endpoint", "arguments": { "path": "customers" } } }
```

After step 6 the endpoint is live—`GET /customers` works immediately without restarting flAPI.

## Error Responses

Config tools return structured errors as a JSON-encoded string inside the standard `-32603` JSON-RPC error message. Always parse the message body for actionable detail:

```
{  "jsonrpc": "2.0",  "id": 1,  "error": {    "code": -32603,    "message": "Tool execution failed: {\"error\":\"Endpoint not found\",\"path\":\"nonexistent\",\"hint\":\"Use flapi_list_endpoints to see available endpoints\"}"  }}
```

The inner JSON typically includes:

*   `error` — short description
*   `path` / `endpoint` / `template` — which resource failed
*   `reason` — underlying exception (when applicable)
*   `hint` — actionable next step for the agent

## Best Practices

1.  **List before mutate.** Always call `flapi_list_endpoints` or `flapi_get_endpoint` before `flapi_update_*` / `flapi_delete_*`.
2.  **Validate templates first.** Call `flapi_test_template` and `flapi_expand_template` before `flapi_update_template`.
3.  **Reload after writes.** Use `flapi_reload_endpoint` to apply YAML changes without a server restart.
4.  **Audit cache changes.** Use `flapi_get_cache_audit` to confirm refreshes landed.
5.  **Bearer tokens, not secrets in prompts.** Pass the config-service token via the bridge's `Authorization` header, not as plaintext arguments.

## See Also

*   **[MCP Overview](/docs/ai-integration/mcp-overview.md)** — what data tools look like
*   **[MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md)** — wire-level JSON-RPC details
*   **[Claude Integration](/docs/ai-integration/claude-integration.md)** — wiring an agent
*   **[Configuration Service](/docs/tools/configuration-service.md)** — the REST API and CLI counterparts of these tools
*   Upstream source of truth: `docs/MCP_CONFIG_TOOLS_API.md` in the [flAPI repository](https://github.com/datazoode/flapi)

([🍪 Cookie Settings](#cookie-settings))

# Model Context Protocol (MCP) Integration

flAPI has built-in support for the **Model Context Protocol (MCP)**—a JSON-RPC 2.0 based standard that lets AI agents like Claude discover and invoke structured tools. With MCP, the same YAML configuration that exposes a REST endpoint can also surface a tool to an LLM, with no duplicate work.

## What is MCP?

The Model Context Protocol is a standardized interface that allows AI agents to:

*   Discover available tools and resources
*   Understand each tool's input schema
*   Call tools with structured arguments
*   Receive structured responses

Think of MCP as OpenAPI/Swagger, but designed specifically for AI agents instead of human developers.

## Why MCP + flAPI?

Traditional approaches require building separate integrations:

*   REST API for applications
*   Custom tool definitions for AI agents
*   Duplicate validation logic
*   Separate documentation

**With flAPI's MCP support:**

*   One YAML file can produce both a REST endpoint and an MCP tool
*   Same validation, same security, same data
*   AI agents get structured access to your enterprise data
*   Endpoint cache, authentication, and rate limits apply consistently

## Architecture in One Picture

```
MCP Client (Claude, Cursor, custom)        │  JSON-RPC 2.0 over HTTP        ▼POST /mcp/jsonrpc        ◄── single transport endpoint        │flAPI MCP Route Handlers        │        ├──► Tools         (mcp-tool YAML blocks)        ├──► Resources     (mcp-resource YAML blocks)        ├──► Prompts       (mcp-prompt YAML blocks)        └──► Config tools  (flapi_* — only with --config-service)        │        ▼DuckDB / Connected sources
```

The MCP transport is exactly one route: `POST /mcp/jsonrpc`. All discovery and invocation flows through that route using JSON-RPC 2.0. A separate `GET /mcp/health` route is available for liveness probes.

## Quick Example

### Define Once, Use Twice

```
# sqls/campaign-stats.yamlurl-path: /campaign-stats# This same file also defines an MCP toolmcp-tool:  name: get_campaign_performance  description: |    Retrieves marketing campaign performance metrics by country.    Returns click counts, revenue, and conversion rates.  result-mime-type: application/jsonrequest:  - field-name: country    field-in: query    description: Two-letter country code (e.g., US, DE, FR)    required: false    validators:      - type: string        regex: "^[A-Z]{2}$"template-source: campaign-stats.sqlconnection:  - bigquery-marketing
```

The supported `mcp-tool` keys are `name`, `description`, and `result-mime-type`. The input schema is generated automatically from `request` fields.

### The SQL Template

```
-- sqls/campaign-stats.sqlSELECT  campaign_type,  country,  SUM(clicks) AS total_clicks,  SUM(revenue) AS total_revenue,  SUM(conversions) AS total_conversions,  ROUND(SUM(revenue) / SUM(clicks), 2) AS revenue_per_clickFROM marketing_campaignsWHERE active = true{{#params.country}}  AND country = '{{{params.country}}}'{{/params.country}}GROUP BY 1, 2ORDER BY total_revenue DESC;
```

### Usage: REST API

```
curl http://localhost:8080/campaign-stats?country=US
```

```
{  "data": [    {      "campaign_type": "social",      "country": "US",      "total_clicks": 150000,      "total_revenue": 45000.00,      "total_conversions": 1200,      "revenue_per_click": 0.30    }  ]}
```

### Usage: AI Agent (MCP)

The agent discovers the tool via `tools/list`:

```
{  "jsonrpc": "2.0",  "id": 1,  "method": "tools/list",  "params": {}}
```

It receives a JSON Schema definition derived from the YAML:

```
{  "name": "get_campaign_performance",  "description": "Retrieves marketing campaign performance metrics by country...",  "inputSchema": {    "type": "object",    "properties": {      "country": {        "type": "string",        "description": "Two-letter country code (e.g., US, DE, FR)"      }    }  }}
```

The agent invokes the tool with `tools/call` (note: the parameter key is `arguments`, not `parameters`):

```
{  "jsonrpc": "2.0",  "id": 2,  "method": "tools/call",  "params": {    "name": "get_campaign_performance",    "arguments": { "country": "US" }  }}
```

The response wraps the SQL result as MCP content blocks:

```
{  "jsonrpc": "2.0",  "id": 2,  "result": {    "content": [      {        "type": "text",        "text": "[{\"campaign_type\":\"social\",\"country\":\"US\",\"total_clicks\":150000, ...}]"      }    ]  }}
```

The AI agent can now reason about campaign performance and make data-driven recommendations.

## MCP Configuration Options

### Basic Tool Definition

```
mcp-tool:  name: tool_name  description: What this tool does
```

### Result MIME Type (Optional)

```
mcp-tool:  name: get_revenue_report  description: Generate a revenue report  result-mime-type: application/json   # default
```

### Per-tool security knobs

```
mcp-tool:  name: customer_lookup  description: Look up a customer by id.  # Required when mcp.auth.enabled: true — flAPI denies every call to a tool  # without an allowed-roles list. The JWT/OIDC principal's `roles` claim  # must intersect this list.  allowed-roles: [analyst, admin]  # Response shaping: applied before the result reaches the agent.  response:    max-rows: 1000                 # hard cap on returned rows    redact-columns: [ssn, salary]  # replace these columns with the redaction sentinel    # sample: true                 # return summary only (row_count, columns, sampled: true)  # Per-tool rate limit, keyed on the authenticated principal (with an  # anonymous fallback bucket per tool).  rate-limit:    enabled: true    max: 30    interval: 60
```

### Shadow / dry-run mode

Append `"_dryRun": true` to the `arguments` object of any `tools/call`:

```
{ "jsonrpc":"2.0","id":3,"method":"tools/call",  "params":{"name":"customer_lookup","arguments":{"id":42,"_dryRun":true}} }
```

flAPI runs validators + template expansion + `EXPLAIN` and returns the rendered SQL + plan as a JSON payload **without** executing the query. RBAC, rate-limit, and validator checks all still apply to dry-runs. Use this for shadow-mode audits before promoting an endpoint to production.

### Tool-description hygiene scanner

When `mcp.strict-descriptions: true` is set in the server config, flAPI refuses to start if any tool's description contains:

*   Control characters or JSON-breakout patterns,
*   Known role-override phrases ("ignore previous instructions", "you are now…", etc.).

This is the in-product defense against prompt-injection via untrusted tool catalogues.

> **Note:** The supported `mcp-tool` keys are `name`, `description`, `result-mime-type`, `allowed-roles`, `response`, and `rate-limit`. The input schema is generated from the endpoint's `request` fields; the description is the primary surface for guiding the LLM.

### MCP Resources

For semi-static data (schemas, reference tables, configuration), use `mcp-resource`:

```
# sqls/customer-schema.yamlmcp-resource:  name: customer_schema  description: Customer database schema definition  mime-type: application/jsontemplate-source: customer-schema.sqlconnection:  - customer-database
```

Resources are addressed by URI (`flapi://customer_schema`) and read via the `resources/read` method.

### MCP Prompts

For reusable prompt templates with Mustache-style placeholders:

```
# sqls/analyze-customer.yamlmcp-prompt:  name: analyze_customer  description: Generate a customer analysis prompt  template: |    Analyze customer {{customer_id}} and produce a churn risk report.  arguments:    - customer_id
```

Clients retrieve a fully substituted prompt via `prompts/get`.

## Real-World Use Cases

### 1\. Customer Support Agent

**Scenario:** AI support agent needs to look up customer information.

```
# Customer lookup toolurl-path: /customers/lookupmcp-tool:  name: lookup_customer  description: Find customer details by email, phone, or IDrequest:  - field-name: email    field-in: query    description: Customer email address  - field-name: phone    field-in: query    description: Customer phone number  - field-name: customer_id    field-in: query    description: Customer ID
```

**Agent Conversation:**

```
User: "I need help with my order, my email is john@example.com"Agent: [calls lookup_customer with email=john@example.com]Agent: "Hi John! I found your account. I see you have 3 orders..."
```

### 2\. Sales Intelligence Agent

**Scenario:** Sales AI that analyzes opportunities.

```
# Opportunity analysisurl-path: /sales/opportunitiesmcp-tool:  name: analyze_sales_pipeline  description: Get sales opportunities with revenue forecastsrequest:  - field-name: stage    description: Pipeline stage (prospecting, negotiation, closing)  - field-name: rep    description: Sales rep name  - field-name: min_value    description: Minimum deal value
```

**Agent Usage:**

```
Sales Manager: "What deals over $100k are in negotiation?"Agent: [calls analyze_sales_pipeline with stage=negotiation, min_value=100000]Agent: "There are 12 deals over $100k in negotiation, totaling $2.4M..."
```

### 3\. Data Analysis Agent

**Scenario:** Claude analyzing business metrics.

```
# Revenue analyticsurl-path: /analytics/revenuemcp-tool:  name: get_revenue_breakdown  description: Revenue analysis by product, region, and time periodrequest:  - field-name: product    description: Product name or ID  - field-name: region    description: Geographic region  - field-name: period    description: Time period (week, month, quarter, year)
```

**Analyst Workflow:**

```
Analyst (to Claude): "Compare Q4 revenue by region vs last year"Claude: [calls get_revenue_breakdown multiple times with different params]Claude: "Here's the comparison: North America is up 23%, Europe is..."
```

## Connecting AI Agents

### Claude Desktop (via stdio bridge)

Claude Desktop's `mcpServers` config currently launches **stdio** subprocesses, but flAPI's MCP server speaks **HTTP** (`POST /mcp/jsonrpc`). To connect Claude Desktop today, run a stdio→HTTP bridge such as [`mcp-remote`](https://www.npmjs.com/package/mcp-remote):

```
// ~/Library/Application Support/Claude/claude_desktop_config.json  (macOS)// %APPDATA%\Claude\claude_desktop_config.json                     (Windows)// ~/.config/Claude/claude_desktop_config.json                     (Linux){  "mcpServers": {    "flapi": {      "command": "npx",      "args": [        "-y",        "mcp-remote",        "http://localhost:8080/mcp/jsonrpc"      ]    }  }}
```

If your flAPI server requires a Bearer token, pass it via `mcp-remote`'s auth header argument or wrap the bridge in a small launcher script. See the [Claude Integration](/docs/ai-integration/claude-integration.md) guide for a complete walkthrough.

> **Note:** Earlier docs suggested `"command": "curl"`. That does not work—Claude Desktop expects a long-lived stdio MCP process, not a one-shot HTTP request. A stdio adapter (such as `mcp-remote` or `mcp-proxy`) is required. Native HTTP-transport support in Claude Desktop is on the MCP roadmap.

### Custom Integration

A direct JSON-RPC client in Python:

```
import requestsBASE = "http://localhost:8080/mcp/jsonrpc"# 1. initializeinit = requests.post(BASE, json={    "jsonrpc": "2.0",    "id": 1,    "method": "initialize",    "params": {        "protocolVersion": "2025-11-25",        "clientInfo": {"name": "py-client", "version": "1.0.0"}    }}).json()session_id = init.get("Mcp-Session-Id")  # also in response headers# 2. list toolstools = requests.post(BASE, json={    "jsonrpc": "2.0",    "id": 2,    "method": "tools/list",    "params": {}}).json()# 3. call a toolresult = requests.post(BASE, json={    "jsonrpc": "2.0",    "id": 3,    "method": "tools/call",    "params": {        "name": "get_campaign_performance",        "arguments": {"country": "US"}    }}).json()
```

For the full message catalog, see the [MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md).

## Security Considerations

### Authentication

MCP requests use the same authentication mechanisms as REST endpoints. flAPI supports Basic, Bearer/JWT, and OIDC authentication on the MCP transport:

```
Authorization: Bearer <jwt-token>
```

Per-method authentication policies allow, for example, an open `tools/list` with an authenticated `tools/call`.

### Row-Level Security

Use the same SQL templates with the authenticated user's `auth.*` context (`auth.username`, `auth.roles`, `auth.email`, `auth.type`, `auth.authenticated`). `auth.roles` is a comma-joined string, so match it with `LIKE`:

```
SELECT * FROM customer_dataWHERE 1=1  AND (    '{{auth.roles}}' LIKE '%admin%'    OR created_by = '{{{auth.username}}}'  )
```

The AI agent inherits the permissions of the authenticated user.

### Per-tool RBAC (deny-by-default)

When `mcp.auth.enabled: true`, each tool's endpoint YAML MUST declare `mcp-tool.allowed-roles`. A tool without an `allowed-roles` list refuses every call — preventing a freshly-added tool from being callable by any authenticated user.

```
mcp-tool:  name: admin_dashboard_metrics  description: KPI rollup for the admin dashboard.  allowed-roles: [admin]            # only callers with `admin` in their JWT roles claim
```

Endpoints without `mcp.auth.enabled` keep working role-free for `flapii project init` demos.

### Rate Limiting

```
# Global rate limit (applies to REST + MCP traffic)rate_limit:  enabled: true  max: 100  interval: 60  key: user-or-ip        # share-NAT-friendly: per-authenticated-user bucket# Per-tool rate limit (declared in the endpoint's mcp-tool block)mcp-tool:  rate-limit:    enabled: true    max: 30    interval: 60
```

### Request audit log

Every REST and MCP call can be recorded to a JSONL file with operator-configurable redaction:

```
audit:  enabled: true  sink: file  path: ./logs/audit.jsonl  redact: [password, api_key]
```

Each event records `{ts, principal, method, target, params, status, row_count, latency_ms}`.

### Startup security auditor

At boot, flAPI scans the loaded config and warns about plaintext passwords, MD5 hashes, MCP exposed without auth on a non-loopback bind, and CORS wildcard combined with `auth.enabled: true`. Combine with `mcp.strict-descriptions: true` for hard-fail behaviour on tool-description prompt-injection patterns.

## Best Practices

### 1\. Write Clear Descriptions

```
# Badmcp-tool:  name: get_data  description: Gets data# Goodmcp-tool:  name: get_customer_orders  description: |    Retrieves all orders for a specific customer, including order status,    items, and total value. Optionally filter by date range.
```

The description is the primary signal an LLM uses when choosing which tool to call—invest in it.

### 2\. Use Meaningful Names

```
# Follow naming conventionsmcp-tool:  name: get_customer_by_id     # clear action + resource  name: search_products         # clear verb  name: calculate_revenue       # descriptive
```

### 3\. Document Each Parameter

```
request:  - field-name: customer_id    description: |      Unique customer identifier. Can be found in the customer      profile or order confirmation email.    required: true
```

The `description` on each request field becomes the JSON Schema property description in the tool's `inputSchema`.

### 4\. Handle Edge Cases

```
SELECT  COALESCE(SUM(revenue), 0) AS total_revenue,  COALESCE(COUNT(*), 0) AS order_countFROM ordersWHERE customer_id = '{{{params.customer_id}}}'  AND date >= CURRENT_DATE - INTERVAL '{{params.days|30}} days';
```

Return meaningful results even when no data exists—LLMs handle empty result sets better than null pointer surprises.

## Debugging MCP Tools

### Health Check

```
curl http://localhost:8080/mcp/health
```

```
{  "status": "healthy",  "server": "flapi-mcp-server",  "version": "0.3.0",  "protocol_version": "2025-11-25",  "mcp_available": true,  "tools_available": true,  "resources_available": true,  "tools_count": 5,  "resources_count": 2}
```

### List Tools via JSON-RPC

```
curl -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | jq
```

### Call a Tool Directly

```
curl -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -d '{    "jsonrpc": "2.0",    "id": 2,    "method": "tools/call",    "params": {      "name": "get_campaign_performance",      "arguments": {"country": "US"}    }  }' | jq
```

For the full set of methods, the session-header model, and error codes, see the [MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md).

## Next Steps

*   **[MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md)**: Full JSON-RPC method catalog
*   **[MCP Config Tools (`flapi_*`)](/docs/ai-integration/mcp-config-tools.md)**: Manage flAPI itself from an AI agent
*   **[Claude Integration Guide](/docs/ai-integration/claude-integration.md)**: Step-by-step Claude Desktop setup
*   **[Endpoints Overview](/docs/endpoints/overview.md)**: Configure API endpoints
*   **[YAML Syntax](/docs/endpoints/yaml-syntax.md)**: Define MCP tool descriptions
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Create dynamic data access
*   **[Authentication](/docs/endpoints/authentication.md)**: Secure your AI tools
*   **[Validation](/docs/endpoints/validation.md)**: Validate AI agent inputs
*   **[BigQuery Example](/docs/examples/bigquery-caching.md)**: See MCP tools in action

Need help building AI-powered data tools? Check out our [professional services](/services).

([🍪 Cookie Settings](#cookie-settings))

# MCP Protocol Reference

This page documents the wire-level protocol that flAPI's MCP server speaks. If you are wiring a client by hand—or debugging an existing one—this is the canonical reference.

For higher-level concepts (why MCP, how YAML maps to tools) see the [MCP Overview](/docs/ai-integration/mcp-overview.md).

## Transport

flAPI exposes MCP as **JSON-RPC 2.0 over HTTP** on a single route:

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/mcp/jsonrpc` | Send any MCP request (initialize, tools, resources, prompts...) |
| `DELETE` | `/mcp/jsonrpc` | Close an MCP session (requires `Mcp-Session-Id` header) |
| `GET` | `/mcp/health` | Plain HTTP liveness probe (not part of MCP) |

The transport is shared with the REST API on the same port — there is no separate MCP port.

### JSON-RPC 2.0 Envelope

Every request and response uses the standard JSON-RPC 2.0 envelope:

**Request:**

```
{  "jsonrpc": "2.0",  "id": 1,  "method": "tools/list",  "params": {}}
```

| Field | Type | Required | Notes |
| --- | --- | --- | --- |
| `jsonrpc` | string | yes | Must be `"2.0"` |
| `id` | string or number | yes | Echoed back in the response. Use `null` only for notifications. |
| `method` | string | yes | One of the methods in the table below |
| `params` | object | no  | Method-specific parameters; defaults to `{}` |

**Success response:**

```
{  "jsonrpc": "2.0",  "id": 1,  "result": { "...": "..." }}
```

**Error response:**

```
{  "jsonrpc": "2.0",  "id": 1,  "error": { "code": -32601, "message": "Method not found" }}
```

### Session Header

flAPI tracks state per MCP session. Sessions are created on the first `initialize` request and identified by the `Mcp-Session-Id` HTTP header.

| Direction | Header | Behaviour |
| --- | --- | --- |
| Response → Client | `Mcp-Session-Id` | Server returns the new (or existing) session id on every response |
| Client → Server | `Mcp-Session-Id` | Echo the value back on subsequent requests to stay in the same session |

Sending no header on the first `initialize` is normal; the server creates a fresh session. To close a session explicitly, send `DELETE /mcp/jsonrpc` with the header set.

## Methods

flAPI implements the following MCP methods. Method names are case-sensitive.

| Method | Purpose | Auth-able |
| --- | --- | --- |
| `initialize` | Handshake, protocol negotiation, capability exchange | yes |
| `ping` | Liveness check inside an MCP session | yes |
| `tools/list` | Discover available tools | yes |
| `tools/call` | Invoke a tool by name with `arguments` | yes |
| `resources/list` | Discover available resources | yes |
| `resources/read` | Read a resource by URI (`flapi://<name>`) | yes |
| `prompts/list` | Discover available prompt templates | yes |
| `prompts/get` | Render a prompt template with arguments | yes |
| `logging/setLevel` | Set server log level for the session | yes |
| `completion/complete` | Auto-complete tool/prompt arguments | yes |

All ten methods are dispatched in `MCPRouteHandlers::dispatchMCPRequest` (`src/mcp_route_handlers.cpp`). Anything else is rejected with `-32601 Method not found`.

### `initialize`

Establishes a session and negotiates the protocol version.

```
{  "jsonrpc": "2.0",  "id": 1,  "method": "initialize",  "params": {    "protocolVersion": "2025-11-25",    "clientInfo": { "name": "my-client", "version": "1.0.0" },    "capabilities": { "sampling": {}, "roots": {} }  }}
```

**Supported protocol versions** (selected via highest mutual match):

*   `2025-11-25` (server default)
*   `2025-06-18`
*   `2025-03-26`
*   `2024-11-05`

If the client requests an unknown version, the server uses its default and logs a warning.

**Response:**

```
{  "jsonrpc": "2.0",  "id": 1,  "result": {    "protocolVersion": "2025-11-25",    "capabilities": {      "tools":     { "listChanged": true },      "resources": { "subscribe": false, "listChanged": true },      "prompts":   { "listChanged": true },      "logging":   {}    },    "serverInfo": { "name": "flapi-mcp-server", "version": "0.3.0" },    "instructions": "..."  }}
```

If the `mcp.instructions` / `instructions-file` setting is configured in `flapi.yaml`, the contents are returned in `result.instructions` for the LLM to read.

### `tools/list`

```
{ "jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {} }
```

Returns an array of tool definitions:

```
{  "tools": [    {      "name": "customer_lookup",      "description": "Retrieve customer information by ID",      "inputSchema": {        "type": "object",        "properties": {          "id": { "type": "string", "description": "Customer ID" }        },        "required": ["id"]      }    }  ]}
```

The schema is generated from the endpoint's `request` fields. All properties currently emit `"type": "string"`; richer JSON Schema typing is a roadmap item.

### `tools/call`

```
{  "jsonrpc": "2.0",  "id": 3,  "method": "tools/call",  "params": {    "name": "customer_lookup",    "arguments": { "id": "12345" }  }}
```

> **Note the parameter shape:** the tool inputs go under `arguments`, **not** `parameters`. This trips up many hand-written clients.

The response wraps the tool result in MCP content blocks:

```
{  "content": [    {      "type": "text",      "text": "[{\"id\":12345,\"name\":\"John Doe\"}]"    }  ]}
```

For SQL-backed tools, the `text` field contains a JSON-serialised array of result rows. For write operations (INSERT/UPDATE/DELETE), it contains `{ "rows_affected": N, "data": ... }`.

**Per-tool RBAC.** When `mcp.auth.enabled: true`, each tool's endpoint YAML MUST declare `mcp-tool.allowed-roles: [admin, analyst]`. Tools without an `allowed-roles` list are denied by default. Calls from a principal whose `roles` claim doesn't intersect the allowed list return `Permission denied: Tool 'X' requires one of [analyst]; caller has [reader].`.

**Per-tool response shaping.** Optionally, the endpoint can declare `mcp-tool.response: { max-rows: 1000, redact-columns: [ssn, salary], sample: false }`. The shaper enforces a hard row cap, replaces listed columns with a redaction sentinel, or returns only summary metadata (`row_count`, `columns`, `sampled: true`) when `sample: true`.

**Per-tool rate limit.** `mcp-tool.rate-limit: { enabled, max, interval }` enforces a per-principal budget for each tool independently of the global rate limiter.

**Dry-run / shadow mode.** Append `"_dryRun": true` to the `arguments` object:

```
{ "jsonrpc":"2.0","id":3,"method":"tools/call",  "params":{"name":"customer_lookup","arguments":{"id":42,"_dryRun":true}}}
```

flAPI runs validators + template expansion + `EXPLAIN` and returns the rendered SQL + plan as a JSON payload **without** executing the query. The payload looks like:

```
{"dry_run": true, "rendered_sql": "SELECT 42 AS customer_id, 'fake' AS name", "params": {"id": "42"}}
```

RBAC, rate-limit, and validator checks all still apply to dry-runs — only the SQL execution step is skipped. Use this for shadow-mode audits before promoting an endpoint to production.

### `resources/list` and `resources/read`

```
{ "jsonrpc": "2.0", "id": 4, "method": "resources/list", "params": {} }
```

Returns resources discovered from `mcp-resource:` YAML blocks. Each resource has a `name`, `description`, `mimeType`, and a URI of the form `flapi://<name>`.

To fetch a resource:

```
{  "jsonrpc": "2.0",  "id": 5,  "method": "resources/read",  "params": { "uri": "flapi://customer_schema" }}
```

Response:

```
{  "contents": [    {      "uri": "flapi://customer_schema",      "mimeType": "application/json",      "text": "{\"fields\":[{\"name\":\"id\",\"type\":\"INTEGER\"}]}"    }  ]}
```

### `prompts/list` and `prompts/get`

`prompts/list` returns prompts declared via `mcp-prompt:`. `prompts/get` substitutes Mustache-style `{{argument_name}}` placeholders and returns a fully rendered chat message:

```
{  "jsonrpc": "2.0",  "id": 6,  "method": "prompts/get",  "params": {    "name": "analyze_customer",    "arguments": { "customer_id": "12345" }  }}
```

Response:

```
{  "description": "Generate customer analysis prompt",  "messages": [    {      "role": "user",      "content": {        "type": "text",        "text": "Analyze customer 12345 and provide insights."      }    }  ]}
```

### `ping`

```
{ "jsonrpc": "2.0", "id": 7, "method": "ping", "params": {} }
```

Returns an empty object `{}` on success — useful for keepalives.

### `logging/setLevel`

Sets the server's log verbosity for diagnostics:

```
{  "jsonrpc": "2.0",  "id": 8,  "method": "logging/setLevel",  "params": { "level": "debug" }}
```

Valid levels: `debug`, `info`, `notice`, `warning`, `error`, `critical`, `alert`, `emergency` (with `notice`→info and `critical`/`alert`/`emergency`→error internally).

### `completion/complete`

Suggests values for a tool or prompt argument. Populated from `enum` validators declared on `request` fields:

```
{  "jsonrpc": "2.0",  "id": 9,  "method": "completion/complete",  "params": { "ref": "customer_lookup", "argument": "status", "value": "act" }}
```

Response:

```
{ "values": ["active"], "total": 1, "hasMore": false }
```

### Notifications

flAPI advertises `listChanged` capability for tools, resources, and prompts in the `initialize` response (the server returns `listChanged: true` for each), but the server does **not** currently push `notifications/tools/list_changed`, `notifications/resources/list_changed`, or `notifications/prompts/list_changed`. Clients that need to detect runtime endpoint mutations from `--config-service` should poll `tools/list` / `resources/list` / `prompts/list` rather than wait for a notification. Client-sent `notifications/initialized` is not recognised by the dispatcher today; if you send one, expect `-32601 Method not found`.

## Session Lifecycle

```
Client                                    flAPI  │                                         │  │ POST /mcp/jsonrpc                       │  │  { method: "initialize", ... }          │  │ ──────────────────────────────────────► │  │                                         │  create session  │ ◄────────────────────────────────────── │  │   Mcp-Session-Id: <sid>                 │  │   { result: { protocolVersion, ... } }  │  │                                         │  │ POST /mcp/jsonrpc                       │  │  Mcp-Session-Id: <sid>                  │  │  { method: "tools/list" }               │  │ ──────────────────────────────────────► │  │ ◄────────────────────────────────────── │  │                                         │  │ DELETE /mcp/jsonrpc                     │  │  Mcp-Session-Id: <sid>                  │  │ ──────────────────────────────────────► │  │                                         │  remove session  │ ◄────────────────────────────────────── │  │   { result: { status: "closed" } }      │
```

Sessions expire after a configurable idle timeout (default 30 minutes). Re-sending `initialize` always creates a fresh session.

## Error Codes

flAPI returns standard JSON-RPC error codes plus two MCP-specific codes. The mapping is in `mcp_route_handlers.cpp` and `mcp_error_builder.cpp`.

| Code | Name | When |
| --- | --- | --- |
| `-32700` | Parse error | Body is not valid JSON |
| `-32600` | Invalid Request | Missing/invalid `method` field |
| `-32601` | Method not found | Method name is not one of the supported methods |
| `-32602` | Invalid params | Required parameter missing or wrong shape (e.g. `tools/call` without `name`) |
| `-32603` | Internal error | Catch-all server-side failure during dispatch |
| `-32001` | Authentication required | Protocol-layer auth (`mcp.auth`) rejected the request |
| `-32000` | Session error | Malformed/missing `Mcp-Session-Id` on session-scoped operation (e.g. `DELETE` without header) |

Error responses include the message and may include context:

```
{  "jsonrpc": "2.0",  "id": 1,  "error": {    "code": -32602,    "message": "Invalid params: missing name"  }}
```

## Health Endpoint

`GET /mcp/health` is a plain HTTP route (not JSON-RPC) for liveness probes and quick smoke tests:

```
curl http://localhost:8080/mcp/health
```

```
{  "status": "healthy",  "server": "flapi-mcp-server",  "version": "0.3.0",  "protocol_version": "2025-11-25",  "mcp_available": true,  "tools_available": true,  "resources_available": true,  "tools_count": 5,  "resources_count": 2,  "arrow_available": true,  "arrow_active_streams": 0,  "arrow_total_requests": 0}
```

Use this in container readiness checks and CI pre-flight steps.

## End-to-End cURL Walkthrough

This is the complete protocol flow against a local flAPI server.

### 1\. Initialize

```
curl -sS -i -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -d '{    "jsonrpc": "2.0",    "id": 1,    "method": "initialize",    "params": {      "protocolVersion": "2025-11-25",      "clientInfo": {"name": "curl-demo", "version": "1.0.0"}    }  }'
```

Capture the `Mcp-Session-Id` response header:

```
HTTP/1.1 200 OKContent-Type: application/jsonMcp-Session-Id: 7a1c-...-9f88
```

Export it for later requests:

```
SID=7a1c-...-9f88
```

### 2\. List tools

```
curl -sS -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -H "Mcp-Session-Id: $SID" \  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' | jq
```

### 3\. Call a tool

```
curl -sS -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -H "Mcp-Session-Id: $SID" \  -d '{    "jsonrpc": "2.0",    "id": 3,    "method": "tools/call",    "params": {      "name": "get_campaign_performance",      "arguments": {"country": "US"}    }  }' | jq
```

### 4\. Close the session

```
curl -sS -X DELETE http://localhost:8080/mcp/jsonrpc \  -H "Mcp-Session-Id: $SID" | jq
```

```
{  "jsonrpc": "2.0",  "id": null,  "result": { "session_id": "7a1c-...-9f88", "status": "closed" }}
```

That is the full lifecycle: handshake, discover, invoke, close.

## See Also

*   **[MCP Overview](/docs/ai-integration/mcp-overview.md)** — concepts and YAML mapping
*   **[Claude Integration](/docs/ai-integration/claude-integration.md)** — wiring Claude Desktop / Claude Code to flAPI
*   **[MCP Config Tools](/docs/ai-integration/mcp-config-tools.md)** — the `flapi_*` admin tool family
*   **[Authentication](/docs/endpoints/authentication.md)** — securing the MCP transport

([🍪 Cookie Settings](#cookie-settings))

# Architecture Deep Dive

This guide explains how flAPI works under the hood, why it's designed the way it is, and how it achieves millisecond API responses while dramatically reducing costs.

**Key Takeaways:**

*   flAPI acts as a **high-performance middleware**, decoupling apps from slow backends.
*   The core is a **DuckDB-powered caching layer** that serves API requests in milliseconds.
*   This results in **99%+ cost savings** and **10,000x faster responses** compared to direct warehouse queries.
*   Caching is **essential** for enterprise systems like SAP and BigQuery.

## The Decoupling Problem

Modern applications and AI agents need fast, frequent access to data. But enterprise data systems weren't built for this:

### Why Data Warehouses Make Poor API Servers

**Snowflake, BigQuery, and SAP are optimized for different workloads:**

|     | Analytical Workload | API Workload |
| --- | --- | --- |
| **Query Pattern** | Few large queries | Many small queries |
| **Latency** | Seconds acceptable | Milliseconds required |
| **Concurrency** | Low (10-100) | High (1000s) |
| **Cost Model** | Pay per data scanned | Pay per query |
| **Frequency** | Periodic (hourly/daily) | Continuous |

**The Cost Problem:**

```
Direct BigQuery Queries:- Single query: $0.05 (scans 10GB)- 1,000 queries/day: $50/day = $1,500/month- 10,000 queries/day: $500/day = $15,000/monthWith flAPI Cache:- Initial query: $0.05- Cache serves 1,000s of queries: $0- Cost: ~$0.05/day = $1.50/month
```

**The Performance Problem:**

*   Warehouse query: 2-10 seconds
*   flAPI cache hit: 0.5-2 milliseconds
*   **1000x to 20,000x faster**

## flAPI's Architecture

flAPI solves this by acting as a high-performance serving layer between your backends and consumers.

### Three-Layer Architecture

### Key Components

#### 1\. API Router

*   Handles incoming HTTP requests
*   Routes to appropriate endpoints
*   Validates parameters
*   Formats responses
*   Serves OpenAPI documentation

#### 2\. Template Engine

*   Processes Mustache templates
*   Injects parameters safely
*   Supports conditional logic
*   Prevents SQL injection

#### 3\. DuckDB Cache Layer

*   In-process analytical database
*   Lightning-fast query execution
*   Scheduled refresh from sources
*   Supports incremental updates

#### 4\. Security Layer

*   JWT authentication
*   Row-level security in SQL
*   Rate limiting per user/endpoint
*   CORS and HTTPS enforcement

#### 5\. Source Connectors

*   Connects to multiple data sources
*   Uses DuckDB's extension ecosystem
*   Pools connections efficiently
*   Handles authentication

## How Caching Works

The cache is the heart of flAPI's performance and cost optimization.

### Cache Lifecycle

```
1. Initial Load   ┌─────────────┐   │  Source DB  │   └──────┬──────┘          │ Expensive query (once)          ▼   ┌─────────────┐   │  DuckDB     │   │  Cache      │   └──────┬──────┘          │          ▼   Ready to serve APIs2. Serving Phase   ┌──────────┐   │ API      │   │ Requests │   └────┬─────┘        │ (1000s of requests)        ▼   ┌─────────────┐   │  DuckDB     │ ← Sub-millisecond responses   │  Cache      │   └─────────────┘3. Refresh   Every N minutes/hours:   ┌─────────────┐   │  Source DB  │ ← Refresh query   └──────┬──────┘          ▼   ┌─────────────┐   │  DuckDB     │ ← Updated data   │  Cache      │   └─────────────┘
```

### Cache Configuration

```
# Example: Marketing campaign data, cached via DuckLakecache:  enabled: true  table: campaign_stats  schema: analytics  schedule: 60m              # Refresh every hour  template-file: campaigns_cache.sql  # Pick the refresh mode by what you provide:  # - omit both -> full refresh  # - cursor only -> incremental append  # - cursor + primary-key -> incremental merge / upsert  primary-key: [campaign_id]  cursor:    column: last_modified    type: timestamp  retention:    keep-last-snapshots: 5    max-snapshot-age: 14d
```

### Cache Refresh Modes

flAPI's cache template is a `SELECT` query — flAPI writes the result set to DuckLake using one of three modes, chosen automatically from the keys above:

**Full refresh** (no `cursor`, no `primary-key`) — replaces all rows in the snapshot.

```
-- campaigns_cache.sqlSELECT *FROM bigquery_scan('project.dataset.campaigns')WHERE active = true
```

**Incremental append** (only `cursor`) — adds rows newer than the previous snapshot.

```
SELECT *FROM bigquery_scan('project.dataset.campaigns')WHERE last_modified > '{{cache.previousSnapshotTimestamp}}'
```

**Incremental merge / upsert** (`cursor` + `primary-key`) — updates changed rows and inserts new ones, keyed on `primary-key`.

```
SELECT *FROM bigquery_scan('project.dataset.campaigns')WHERE last_modified > '{{cache.previousSnapshotTimestamp}}'
```

## Query Flow

### Without flAPI (Direct Warehouse Access)

```
User Request → Application → BigQuery → Wait 3s → Response                             ($0.05)
```

Every request hits the warehouse. Slow and expensive.

### With flAPI (Cached Serving)

```
User Request → flAPI → DuckDB Cache → Response (2ms)                       (free)Background:Every hour → flAPI → BigQuery → Refresh Cache                    ($0.05)
```

Thousands of requests served from one warehouse query.

## Performance Characteristics

### Latency Comparison

| Data Source | Latency | Use Case |
| --- | --- | --- |
| BigQuery (direct) | 2-10s | Analytical queries |
| Snowflake (direct) | 1-5s | Data warehouse queries |
| Postgres (direct) | 50-500ms | Transactional queries |
| flAPI (cached) | 1-50ms | API serving |
| **Improvement** | **1000-20,000x** |     |

### Cost Comparison

**Scenario: Analytics API with 10,000 queries/day**

| Approach | Daily Cost | Monthly Cost | Annual Cost |
| --- | --- | --- | --- |
| Direct BigQuery | $500 | $15,000 | $180,000 |
| flAPI (hourly refresh) | $1.20 | $36 | $432 |
| **Savings** | **99.76%** | **99.76%** | **99.76%** |

## Scaling Patterns

### Vertical Scaling

flAPI is single-threaded but incredibly efficient. For most use cases, a single instance handles:

*   **10,000+ requests/second** (cached queries)
*   **Millions of rows** in cache
*   **Dozens of endpoints**

Increase instance size for:

*   Larger cache datasets
*   More complex SQL transforms
*   Higher concurrent connections

### Horizontal Scaling

For massive scale, run multiple flAPI instances:

```
Load Balancer    ├── flAPI Instance 1 (with cache)    ├── flAPI Instance 2 (with cache)    └── flAPI Instance 3 (with cache)
```

Each instance has its own cache (eventually consistent). Cache refresh happens independently.

### Serverless Deployment

flAPI's millisecond startup makes it perfect for serverless:

```
# AWS Lambda Example- Memory: 1GB- Startup: < 100ms cold start- Execution: 1-5ms per request- Cost: Pay only for requests
```

Cache can be stored in:

*   EFS (shared across invocations)
*   S3 + local temp (reload on cold start)
*   External database (DuckDB over HTTP)

## Security Architecture

### Multi-Layer Security

```
1. Network Layer   ├── HTTPS enforcement   ├── CORS policies   └── Rate limiting2. Authentication   ├── JWT tokens   ├── API keys   └── Custom auth hooks3. Authorization   ├── Endpoint-level access   ├── Row-level security in SQL   └── Column-level filtering4. Data Protection   ├── SQL injection prevention   ├── Input validation   └── Output sanitization
```

### Row-Level Security Example

```
-- Endpoint SQL template-- auth.roles is a comma-joined string (e.g. "read,admin"), so we match with LIKE.SELECT *FROM sensitive_dataWHERE 1=1  AND (    -- Admins see everything    '{{auth.roles}}' LIKE '%admin%'    OR (      -- Other users see only their own region / department      region = '{{{params.region}}}'      AND created_by = '{{{auth.username}}}'    )  )
```

The cache contains all data, but the serving layer filters based on the authenticated user's `auth.*` context (`auth.username`, `auth.roles`, `auth.email`, `auth.type`, `auth.authenticated`).

## When to Use flAPI

### Ideal Use Cases ✅

*   **High-frequency access to stable data**: Dashboard APIs, analytics endpoints
*   **AI agent tools**: Structured data access for LLMs
*   **Customer-facing APIs**: Fast responses required
*   **Cost-sensitive workloads**: Reduce warehouse bills
*   **Multi-consumer scenarios**: Multiple apps accessing same data

### Not Ideal For ❌

*   **Real-time transactional data**: Use a transactional database instead
*   **Data that changes every second**: Cache refresh overhead too high
*   **Single-query workloads**: No benefit from caching
*   **Extremely large datasets**: Cache may not fit in memory

### Hybrid Approach

Use flAPI for frequently-accessed data and direct queries for one-off analytics:

```
┌─────────────────┐│   Application   │└────────┬────────┘         │    ┌────┴────┐    │         │    ▼         ▼┌────────┐ ┌──────────┐│ flAPI  │ │ Snowflake││ (fast) │ │ (direct) │└────────┘ └──────────┘  │             │  └─────┬───────┘        │  ┌─────▼──────┐  │  Warehouse │  └────────────┘
```

## Next Steps

*   **[How It Works](/docs/concepts/how-it-works.md)**: Simple 3-step process overview
*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: Understand cost optimization through caching
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Master dynamic query patterns
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Configure caching for your endpoints
*   **[Endpoints Overview](/docs/endpoints/overview.md)**: Learn about endpoint configuration
*   **[Authentication](/docs/endpoints/authentication.md)**: Secure your APIs
*   **[Deployment Guide](/docs/getting-started/deployment.md)**: Deploy to production
*   **[Examples](/docs/examples/bigquery-caching.md)**: See architecture in action

Need help architecting your solution? Check out our [professional services](/services).

([🍪 Cookie Settings](#cookie-settings))

# Caching Strategy

flAPI's caching layer is built on **DuckLake** — a snapshot-based table format over DuckDB. It is the flagship caching feature in flAPI and provides far more than a TTL cache: every refresh produces an immutable snapshot you can roll back to, expire on a retention policy, or use as a checkpoint for incremental merges.

> Think of DuckLake caching like a **versioned ledger**: every refresh appends a new snapshot, the latest snapshot serves the API in milliseconds, and old snapshots are pruned by a retention policy you control.

## Why DuckLake?

Data warehouses like BigQuery, Snowflake, and SAP are optimized for analytical workloads, not API serving:

| Requirement | Warehouse | API Serving |
| --- | --- | --- |
| **Latency** | 2-10 seconds | < 5 ms |
| **Concurrency** | Low (10-100) | High (1000s) |
| **Query Pattern** | Few large queries | Many small queries |
| **Frequency** | Periodic | Continuous |

DuckLake caching gives you warehouse-quality SQL semantics with serving-tier latency, plus:

*   **Snapshot isolation** — readers always see a consistent table version.
*   **Time travel** — query the cache as of any retained snapshot.
*   **Three refresh modes** — full / append / merge, chosen automatically from your config.
*   **Retention & expiry** — drop old snapshots by count or age.

## The Global DuckLake Config

Caching is enabled per endpoint, but the DuckLake catalog itself is configured once in `flapi.yaml`:

```
# flapi.yamlducklake:  enabled: true  alias: cache                         # Catalog alias used in cache templates  metadata-path: ./data/cache.ducklake # DuckLake metadata directory  data-path: ./data/cache              # DuckLake data files directory  retention:    keep-last-snapshots: 10    max-snapshot-age: 30d  compaction:    enabled: true    schedule: '@daily'  scheduler:    enabled: true    scan-interval: 5m                  # How often the scheduler checks endpoints
```

Without a `ducklake:` block, endpoint-level caching is inert.

## Per-Endpoint Cache Configuration

A cache is declared on the endpoint YAML. All keys are real and verified against `cache_manager.cpp`:

```
# sqls/customers/customers-rest.yamlcache:  enabled: true  table: customers_cache  schema: analytics                # optional, default "main"  schedule: 5m                     # 30s | 5m | 1h | 2d  primary-key: [id]                # required for merge mode  cursor:    column: registration_date      # required for append / merge modes    type: date                     # int | date | timestamp  rollback-window: 2d  retention:    keep-last-snapshots: 5    max-snapshot-age: 14d  delete-handling: soft            # soft | hard  template-file: customers_cache.sql   # optional custom refresh SQL
```

## The Three Refresh Modes

flAPI does not have a `strategy:` key. The refresh mode is **derived** from the combination of `cursor` and `primary-key`, per `CacheManager::determineCacheMode` in `cache_manager.cpp`:

| `cursor` | `primary-key` | Mode | What happens |
| --- | --- | --- | --- |
| absent | _any_ | **full** | Cache table is rebuilt from scratch each refresh |
| present | absent | **append** | New rows past `cursor.column` are appended |
| present | present | **merge** | Inserts, updates and deletes reconciled by primary key + cursor |

### Full Refresh

Used when there is no `cursor`. The cache template runs a `CREATE OR REPLACE TABLE` against the DuckLake catalog. Good for small reference tables.

```
cache:  enabled: true  table: countries_cache  schedule: 1h
```

### Incremental Append

Add `cursor` and the engine switches to append mode. Best for append-only event-style data:

```
cache:  enabled: true  table: events_cache  schedule: 5m  cursor:    column: occurred_at    type: timestamp
```

### Incremental Merge

Add both `cursor` and `primary-key`. flAPI merges inserts, updates and deletes:

```
cache:  enabled: true  table: customers_cache  schedule: 1m  primary-key: [id]  cursor:    column: updated_at    type: timestamp
```

`delete-handling: soft` tombstones removed rows; `hard` deletes them physically.

## Cache Template Variables

When a refresh runs, flAPI renders the cache SQL through the SQL template processor and injects a `cache.*` context. Verified against `sql_template_processor.cpp`:

| Variable | Description |
| --- | --- |
| `{{cache.catalog}}` | DuckLake catalog alias from `ducklake.alias` |
| `{{cache.schema}}` | Cache schema (default `main`) |
| `{{cache.table}}` | Cache table name |
| `{{cache.schedule}}` | Configured refresh schedule |
| `{{cache.mode}}` | Refresh mode: `full`, `append`, or `merge` |
| `{{cache.snapshotId}}` | Current snapshot ID |
| `{{cache.snapshotTimestamp}}` | Current snapshot timestamp |
| `{{cache.previousSnapshotId}}` | Previous snapshot ID (for incremental) |
| `{{cache.previousSnapshotTimestamp}}` | Previous snapshot timestamp |
| `{{cache.cursorColumn}}` | `cursor.column` value |
| `{{cache.cursorType}}` | `cursor.type` value |
| `{{cache.primaryKeys}}` | Comma-separated primary key columns |

### Example: DuckLake Merge Template

```
-- sqls/customers/customers_cache.sqlCREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} ASSELECT    id,    name,    email,    segment,    registration_date,    CURRENT_TIMESTAMP AS cache_updated_at,    '{{cache.snapshotId}}' AS cache_snapshot_idFROM read_parquet('{{conn.path}}')ORDER BY registration_date DESC;
```

The example above is the real merge-mode template shipped with the customers example (`examples/sqls/customers/customers_cache.sql`).

## Cost Math

A typical 50,000-requests/day workload against BigQuery (10 GB scan @ $0.05/query):

| Refresh schedule | Daily cost | Monthly cost |
| --- | ---: | ---: |
| Every 5 minutes | $14.40 | $432 |
| Every 15 minutes | $4.80 | $144 |
| Every hour | $1.20 | $36 |
| Every 6 hours | $0.20 | $6  |
| Daily | $0.05 | $1.50 |

Direct access for the same workload costs $2,500/day. Caching trades freshness for cost in a predictable, snapshot-aware way.

## Retention, Rollback & Audit

DuckLake snapshots are first-class. flAPI exposes:

*   **`retention.keep-last-snapshots`** — keep N most recent snapshots per table.
*   **`retention.max-snapshot-age`** — drop snapshots older than e.g. `30d`.
*   **`rollback-window`** — keep snapshots queryable for time-travel during this window.
*   **Audit table** — `<catalog>.audit.sync_events` records every refresh with `sync_type`, `status`, `message`, timestamps and duration. Created automatically by `CacheManager::initializeAuditTables`.

Expiry is implemented by calling `ducklake_expire_snapshots(...)` against the catalog.

## When to Use DuckLake Caching

### Ideal

*   High-frequency reads of moderately fresh data (dashboards, AI tool calls).
*   Backends with per-query cost (BigQuery, Snowflake, SAP).
*   Workloads that benefit from snapshot isolation (audit, reproducibility).

### Not Ideal

*   Sub-second freshness — refresh overhead is too high.
*   Single-query workloads — no amortization.
*   Datasets that don't fit on the flAPI host's local disk.

## Next Steps

*   **[Caching Setup Guide](/docs/guides/caching/setup.md)**: Step-by-step DuckLake configuration
*   **[How It Works](/docs/concepts/how-it-works.md)**: Understand the 3-step flAPI process
*   **[Architecture Deep Dive](/docs/concepts/architecture.md)**: Technical details of the caching layer
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Cache and API templates
*   **[BigQuery Example](/docs/examples/bigquery-caching.md)**: Complete caching implementation

([🍪 Cookie Settings](#cookie-settings))

# How It Works

flAPI simplifies the process of creating data APIs into three straightforward steps. This effectively decouples high-throughput API consumers (like AI agents or web services) from traditional backend data systems, resulting in significantly improved performance and lower costs.

## The Three-Step Process

### 1\. Connect Your Data

Point flAPI to your data sources—whether they're cloud data warehouses like BigQuery and Snowflake, traditional databases like PostgreSQL, or file formats like Parquet and CSV.

```
# Example: Connect to multiple data sourcesconnections:  bigquery-warehouse:    init: |      INSTALL 'bigquery';      LOAD 'bigquery';    properties:      project_id: 'my-project'        customers-parquet:    properties:      path: './data/customers.parquet'
```

**What happens:** flAPI leverages DuckDB's powerful extension ecosystem to connect to 20+ data sources out of the box. You can mix and match sources in a single API.

### 2\. Define APIs with SQL

Write standard SQL queries and define your endpoints using simple YAML files. Use Mustache syntax to create dynamic, parameterized queries.

**Endpoint Configuration** (`sqls/customers.yaml`):

```
url-path: /customers/request:  - field-name: segment    field-in: query    description: Filter by market segment    required: falsetemplate-source: customers.sqlconnection:  - customers-parquet
```

**SQL Template** (`sqls/customers.sql`):

```
SELECT  c_custkey as id,  c_name as name,  c_acctbal as balanceFROM '{{{conn.path}}}'WHERE 1=1{{#params.segment}}  AND c_mktsegment LIKE '%{{{params.segment}}}%'{{/params.segment}}
```

**What happens:** flAPI automatically handles parameter validation, SQL injection prevention, and generates OpenAPI documentation.

### 3\. Access Instantly

Run the flAPI server and immediately access your data through secure, documented REST APIs—plus MCP tools for AI agents.

```
# Start flAPI$ ./flapi -c config.yaml✓ Loaded 12 endpoints✓ Server listening on :8080⚡ Ready in 1.2ms
```

```
# Call your API$ curl http://localhost:8080/customers?segment=AUTO{  "data": [    {"id": 1, "name": "Customer ABC", "balance": 7500.25},    {"id": 2, "name": "Customer XYZ", "balance": 12300.50}  ]}
```

**What happens:** Your data is now accessible via a production-ready REST API with authentication, rate limiting, and automatic documentation.

## The Architecture

## Why This Matters

### Problem: Direct Warehouse Access

```
User Request → BigQuery → Wait 2-10s → $0.05 cost(Every single request hits the warehouse)10,000 requests/day = $500/day = $15,000/month
```

### Solution: flAPI Serving Layer

```
User Request → flAPI Cache → Response in 1-50ms → Free(Warehouse queried once per hour)Background: flAPI → BigQuery → Refresh cache ($0.05)Total cost: ~$1.20/day = $36/monthSavings: 99.76%Speed: 1000-20,000x faster
```

## Key Benefits

*   **🚀 Performance**: Sub-millisecond responses instead of seconds
*   **💰 Cost Reduction**: 90%+ savings on warehouse costs
*   **🔒 Security**: Built-in authentication, rate limiting, row-level security
*   **🤖 AI-Ready**: Automatic MCP tool generation for AI agents
*   **📦 Simple**: Single binary, zero dependencies, local-first development

## Next Steps

*   **[Architecture Deep Dive](/docs/concepts/architecture.md)**: Understand the technical details and performance characteristics
*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: Learn how caching saves costs and improves performance
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Master dynamic SQL queries with Mustache
*   **[Quickstart Guide](/docs/getting-started/quickstart.md)**: Build your first API in 5 minutes
*   **[Endpoints Overview](/docs/endpoints/overview.md)**: Configure REST API endpoints
*   **[Examples](/docs/examples/parquet-api.md)**: See real-world implementations

([🍪 Cookie Settings](#cookie-settings))

# SQL Templating

flAPI uses **Mustache templating** to create dynamic, parameterized SQL queries. This approach, inspired by dbt, is immediately familiar to data analysts and data scientists while providing powerful flexibility.

## How Template Processing Works

## Why Templates?

Instead of writing separate SQL for every possible query combination, write one template that adapts:

```
-- Without templates: many queriesSELECT * FROM customers WHERE segment = 'AUTOMOBILE';SELECT * FROM customers WHERE segment = 'BUILDING';SELECT * FROM customers WHERE segment = 'FURNITURE' AND id = 42;-- ... hundreds of combinations-- With templates: one querySELECT * FROM customersWHERE 1=1{{#params.segment}}  AND segment = '{{{ params.segment }}}'{{/params.segment}}{{#params.id}}  AND id = {{ params.id }}{{/params.id}}
```

## Basic Syntax

### Variable Insertion

Use `{{ }}` (double braces) for **typed equality lookups** (the value is bound as a DuckDB prepared-statement parameter) and `{{{ }}}` (triple braces) for **substring patterns, identifier insertion, and other places where the value participates in the SQL text**:

```
-- Typed scalar (validator type: int/double/bool/date/time/uuid/enum/email/string)-- The renderer emits a `?` placeholder and binds the value via duckdb_bind_*.-- SQL injection at this site is structurally impossible.WHERE customer_id = {{ params.id }}-- Substring / LIKE / non-equality — value flows through Mustache as text,-- single quotes required around itWHERE name LIKE '%{{{ params.name }}}%'
```

When `params.id = 12345` and `params.name = "Acme"`, the prepared form renders as:

```
WHERE customer_id = ?     -- bound: int64(12345)WHERE name LIKE '%Acme%'
```

flAPI's SQL-injection defense is **layered**:

1.  **`RequestValidator`** rejects malformed inputs (typed fields, range/regex/enum/format checks). Integer, date, and time parsing is strict — `1; DROP TABLE` no longer slips through as `1`, and `2024-03-15' OR 1=1` no longer slips through as `2024-03-15`.
2.  **DuckDB prepared-statement bind** is the hard boundary for typed double-brace references on **every code path** — GET, POST, PUT, PATCH, Arrow-streaming endpoints, and multi-statement INSERT…;SELECT…RETURNING templates. The value travels as a primitive, not text, and cannot smuggle SQL.
3.  **Keyword regex fallback** still rejects obvious injection patterns for fields where the prepared path can't apply (triple-brace sites and untyped fields). It's demoted to a debug-level log only for numeric/temporal bindable fields where the regex is a known false-positive source.

If a typed parameter cannot be converted to its SQL type (e.g. `id=abc` for an int field), flAPI returns **HTTP 400** with a JSON error body — bind-conversion failures are client input errors, not server errors.

End-to-end injection corpora ship in the repo:

*   [`test_sql_injection_corpus.py`](https://github.com/DataZooDE/flapi/blob/main/test/integration/test_sql_injection_corpus.py) — 99 GET-path payloads across all nine validator types plus a pagination-with-bindings endpoint.
*   [`test_sql_injection_write_corpus.py`](https://github.com/DataZooDE/flapi/blob/main/test/integration/test_sql_injection_write_corpus.py) — 19 POST-path payloads including multi-statement `INSERT … ; SELECT … RETURNING` templates.

Both confirm every classic injection pattern (UNION, OR 1=1, comment-evasion, xkcd 327) either returns zero rows or is rejected at the validator/bind boundary — none execute as SQL.

### Conditional Blocks

Use `{{#variable}}...{{/variable}}` for conditional inclusion:

```
SELECT * FROM ordersWHERE 1=1{{#params.status}}  AND status = '{{{ params.status }}}'{{/params.status}}{{#params.start_date}}  AND order_date >= '{{{ params.start_date }}}'{{/params.start_date}}
```

**When params are provided** (`{"status": "completed", "start_date": "2024-01-01"}`):

```
SELECT * FROM ordersWHERE 1=1  AND status = 'completed'  AND order_date >= '2024-01-01'
```

**When params are empty** (`{}`):

```
SELECT * FROM ordersWHERE 1=1
```

### Inverted Sections (Unless)

Use `{{^variable}}...{{/variable}}` for "if not" conditions:

```
SELECT * FROM productsWHERE 1=1{{^params.include_discontinued}}  AND active = true{{/params.include_discontinued}}
```

## Context Variables

flAPI exposes the following Mustache contexts to every SQL template (defined in `src/sql_template_processor.cpp`):

### Request Parameters (`params.*`)

Access validated query, path, header and body parameters:

```
SELECT * FROM customersWHERE 1=1{{#params.segment}}  AND market_segment = '{{{ params.segment }}}'{{/params.segment}}{{#params.id}}  AND id = {{ params.id }}{{/params.id}}
```

Path parameters declared with `:name` in `url-path` are available the same way (`params.id` for `/customers/:id`).

### Connection Properties (`conn.*`)

Access properties of the first connection listed under `connection:` in the endpoint YAML:

```
connections:  customers-parquet:    properties:      path: './data/customers.parquet'
```

```
SELECT * FROM '{{{ conn.path }}}'WHERE active = true
```

### Authentication Context (`auth.*`)

When an authenticated request hits the endpoint, the following fields are populated:

| Field | Description |
| --- | --- |
| `auth.username` | The authenticated principal's username |
| `auth.roles` | The principal's roles (space-separated string suitable for matching) |
| `auth.email` | The principal's email (when available, e.g., OIDC) |
| `auth.type` | Authentication scheme: `basic`, `jwt`, `bearer`, or `oidc` |
| `auth.authenticated` | `"true"` when the request was authenticated |

Use these for row-level security or audit logging:

```
SELECT *FROM sales_dataWHERE 1=1{{#auth.authenticated}}  AND sales_rep = '{{{ auth.username }}}'{{/auth.authenticated}}
```

To branch on a specific role, perform the check inside the SQL itself (Mustache iterates `auth.roles` as a string, not a set):

```
SELECT *FROM ordersWHERE 1=1  AND (    -- Admin sees everything; otherwise restrict to own region    '{{{ auth.roles }}}' LIKE '%admin%'    OR sales_rep = '{{{ auth.username }}}'  )
```

### Cache Properties (`cache.*`)

In cache-refresh templates only, these fields are available:

| Field | Description |
| --- | --- |
| `cache.catalog` | DuckLake catalog name |
| `cache.schema` | Cache schema |
| `cache.table` | Cache table |
| `cache.schedule` | Refresh schedule string |
| `cache.previousSnapshotId` / `cache.previousSnapshotTimestamp` | Previous snapshot pointers |
| `cache.cursorColumn` / `cache.cursorType` / `cache.primaryKeys` | Incremental refresh helpers |

### Environment Variables (`env.*`)

Whitelisted environment variables (see `template.environment-whitelist` in `flapi.yaml`):

```
SELECT *FROM logsWHERE region = '{{{ env.AWS_REGION }}}'
```

## Advanced Patterns

### Dynamic Column Selection

```
SELECT  customer_id,  name  {{#params.include_pii}}  , email  , phone  {{/params.include_pii}}FROM customers
```

### Conditional Joins

```
SELECT  o.order_id,  o.total  {{#params.include_customer}}  , c.customer_name  {{/params.include_customer}}FROM orders o{{#params.include_customer}}LEFT JOIN customers c ON o.customer_id = c.id{{/params.include_customer}}
```

### Default Values via Inverted Sections

```
SELECT * FROM productsORDER BY {{#params.sort_by}}{{{ params.sort_by }}}{{/params.sort_by}}{{^params.sort_by}}created_at{{/params.sort_by}}LIMIT {{#params.limit}}{{ params.limit }}{{/params.limit}}{{^params.limit}}100{{/params.limit}}
```

## Security Considerations

### SQL Injection Prevention

flAPI prevents SQL injection through **parameter validation** (`src/request_validator.cpp`):

```
request:  - field-name: status    field-in: query    validators:      - type: enum        allowedValues: ['pending', 'completed', 'cancelled']
```

Every parameter is type-checked, range-checked, and (by default) scanned for SQL-injection patterns before it ever reaches the template engine.

### Whitelist Pattern

Always restrict free-form text inputs with a validator:

```
# Good: enum whitelistvalidators:  - type: enum    allowedValues: ['name', 'email', 'created_at']# Good: strict regexvalidators:  - type: string    regex: '^[A-Za-z_]+$'    preventSqlInjection: true
```

### Safe Variable Forms

```
-- Integers/identifiersWHERE customer_id = {{ params.id }}-- Strings (always single-quoted around triple-brace)WHERE name = '{{{ params.name }}}'
```

## Real-World Examples

### Example 1: Filterable Customer API

**Endpoint** (`sqls/customers.yaml`):

```
url-path: /customers/method: GETrequest:  - field-name: segment    field-in: query    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY]  - field-name: min_balance    field-in: query    required: false    validators:      - type: int        min: 0        max: 1000000template-source: customers.sqlconnection:  - customers-parquet
```

**Template** (`sqls/customers.sql`):

```
SELECT  c_custkey      AS id,  c_name         AS name,  c_mktsegment   AS segment,  c_acctbal      AS balanceFROM '{{{ conn.path }}}'WHERE 1=1{{#params.segment}}  AND c_mktsegment = '{{{ params.segment }}}'{{/params.segment}}{{#params.min_balance}}  AND c_acctbal >= {{ params.min_balance }}{{/params.min_balance}}ORDER BY c_acctbal DESCLIMIT 100
```

### Example 2: Row-Level Security via `auth.*`

```
SELECT  order_id,  customer_name,  total,  statusFROM ordersWHERE 1=1  AND (    '{{{ auth.roles }}}' LIKE '%admin%'    OR sales_rep = '{{{ auth.username }}}'  ){{#params.status}}  AND status = '{{{ params.status }}}'{{/params.status}}
```

### Example 3: Cache Materialization

**Cache Template** (`sqls/analytics_cache.sql`):

```
CREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} ASSELECT  DATE(order_date)  AS date,  product_category,  country,  COUNT(*)          AS order_count,  SUM(revenue)      AS total_revenueFROM bigquery_scan('project.dataset.orders')WHERE order_date >= CURRENT_DATE - INTERVAL '90 days'GROUP BY 1, 2, 3
```

## Best Practices

### 1\. Always Start with `WHERE 1=1`

Makes conditional filters trivial to add or remove:

```
WHERE 1=1{{#params.filter1}}  AND field1 = '{{{ params.filter1 }}}'{{/params.filter1}}
```

### 2\. Validate Every Parameter

```
validators:  - type: string    regex: '^[A-Z]{2}$'    preventSqlInjection: true
```

### 3\. Match Brace Form to Type

*   Integer/numeric parameters: `{{ params.x }}` (double).
*   String/date/email/uuid parameters: `'{{{ params.x }}}'` (triple, quoted).

### 4\. Comment Your Templates

```
-- Customer API template-- Supports filtering by segment and minimum balanceSELECT ...
```

### 5\. Keep Templates Focused

One template per endpoint. Don't try to encode many use cases in a single file.

## Next Steps

*   **[Quickstart Guide](/docs/getting-started/quickstart.md)**: Build your first templated API
*   **[How It Works](/docs/concepts/how-it-works.md)**: Understand the complete flAPI process
*   **[Endpoint Configuration](/docs/endpoints/overview.md)**: Learn endpoint setup
*   **[Parameters](/docs/endpoints/parameters.md)**: Define and access request parameters
*   **[Validation](/docs/endpoints/validation.md)**: Validate parameters safely
*   **[YAML Syntax](/docs/endpoints/yaml-syntax.md)**: Advanced YAML features and includes

([🍪 Cookie Settings](#cookie-settings))

# Authentication

flAPI exposes three `auth.type` values, evaluated by the auth middleware in `auth_middleware.cpp:164-171`:

| `type` | Use case | Notes |
| --- | --- | --- |
| `basic` | HTTP Basic — inline users or AWS-Secrets-backed user tables | Verify in `auth_middleware.cpp` (`processBasicAuth`); user list either inline under `auth.users:` or loaded from AWS via `auth.from-aws-secretmanager` |
| `bearer` | Stateless JWT bearer tokens (HS256) | Set `jwt-secret` and `jwt-issuer` to validate the token; same code path also recognises shared-secret bearer tokens |
| `oidc` | OpenID Connect via Google, Microsoft, Keycloak, Auth0, Okta, GitHub, or a generic provider | Set `auth.oidc.provider-type` to one of `google`, `microsoft`, `keycloak`, `auth0`, `okta`, `github`, or `generic` (verified against `oidc_provider_presets.cpp`) |

There are no other `auth.type` values. There is no `type: jwt` (use `type: bearer` with `jwt-secret`), no "API key" header (`X-API-Key`) flow, no custom handler plug-in, and no role-defining config block. JWT and AWS Secrets Manager are not separate schemes — they are configuration options that combine with the three real `type` values above.

## Configuration Key

All authentication is configured under the top-level `auth:` key (not `security:` or `authentication:`).

```
auth:  enabled: true  type: basic            # basic | bearer | oidc  # ...scheme-specific keys
```

## Global vs Per-Endpoint Auth

`auth` can be set globally in `flapi.yaml` and overridden per endpoint YAML. An endpoint-level `auth:` block fully replaces the global one for that endpoint.

```
# flapi.yamlauth:  enabled: false        # disabled globally
```

```
# sqls/customers/customers-rest.yamlurl-path: /customers/auth:                   # per-endpoint override  enabled: true  type: bearer  jwt-secret: '${JWT_SECRET}'  jwt-issuer: my-auth-server
```

When authentication fails, flAPI returns `401 Unauthorized` with the `WWW-Authenticate: Basic realm="flAPI"` header.

## Basic Authentication

Inline users with hashed passwords. flAPI auto-detects three formats by prefix:

| Format | Detected by | Status |
| --- | --- | --- |
| `$pbkdf2-sha256$<iter>$<b64-salt>$<b64-hash>` | leading `$pbkdf2-sha256$` | **Recommended.** Modular Crypt Format, compatible with Python `passlib` and other PBKDF2-SHA256 generators. flAPI uses OpenSSL `PKCS5_PBKDF2_HMAC` with 600 000 iterations (OWASP 2023), 16-byte salt, 32-byte derived key |
| 32-character lowercase hex | length+charset | MD5 hash. Still accepted, but the **startup auditor** emits a deprecation warning at boot — MD5 has no salt and is fast to brute-force |
| anything else | fallback | Plaintext. Accepted for local demos only; the startup auditor warns about it |

```
auth:  enabled: true  type: basic  users:    # Recommended: PBKDF2-SHA256 hash. Generate with `passlib`:    #   from passlib.hash import pbkdf2_sha256    #   print(pbkdf2_sha256.using(rounds=600000).hash("secret"))    - username: admin      password: '$pbkdf2-sha256$600000$saltsaltsaltsalt$baseekey...'      roles: [admin, read, write]    - username: '{{env.CUSTOMER_API_READ_USER}}'      password: '{{env.CUSTOMER_API_READ_PASSWORD}}'      roles: [read]
```

Usage:

```
curl -u admin:secret https://api.example.com/customers/
```

## Bearer (JWT) Authentication

`type: bearer` validates `Authorization: Bearer <token>` headers. When `jwt-secret` is set, the token is parsed as an HS256-signed JWT and the signature is verified against the secret; otherwise the bearer string is treated as an opaque shared secret. `jwt-issuer` further constrains accepted tokens to a specific issuer.

```
auth:  enabled: true  type: bearer  jwt-secret: '${JWT_SECRET}'  jwt-issuer: my-auth-server
```

Expected JWT payload:

```
{  "sub": "user123",  "iss": "my-auth-server",  "roles": ["user", "admin"],  "exp": 1735689600}
```

*   `sub` is extracted into `auth.username`
*   `roles` (a JSON array) is extracted into `auth.roles` (joined as a comma-separated string in the Mustache context)

Usage:

```
curl -H "Authorization: Bearer $TOKEN" https://api.example.com/customers/
```

## OIDC Authentication

OpenID Connect with JWKS-based asymmetric signature validation, claim mapping, and provider presets. Configured under `auth.oidc.*`.

### Provider Presets

flAPI ships with presets that fill in sensible defaults (issuer URL templates, scopes, claim mappings):

| `provider-type` | Issuer template | Default `username-claim` | Notes |
| --- | --- | --- | --- |
| `google` | `https://accounts.google.com` | `email` | Workspace SSO |
| `microsoft` | `https://login.microsoftonline.com/{tenant}/v2.0` | `preferred_username` | `{tenant}` required |
| `keycloak` | `https://keycloak.example.com/realms/{realm}` | `preferred_username` | `{realm}` required; roles in `realm_access.roles` |
| `auth0` | `https://{domain}.auth0.com` | `email` | `{domain}` required |
| `okta` | `https://{domain}.okta.com/oauth2/default` | `preferred_username` | `{domain}` required |
| `github` | `https://github.com` | `login` | OAuth 2.0 (not full OIDC) |
| `generic` | _(must be provided)_ | `sub` | Any spec-compliant OIDC IdP |

### Full Key Reference

| Key | Default | Description |
| --- | --- | --- |
| `auth.oidc.issuer-url` | _(required for `generic`)_ | OIDC discovery base URL |
| `auth.oidc.client-id` | _(required)_ | OAuth client ID |
| `auth.oidc.client-secret` | \-  | OAuth client secret (refresh / client-credentials flows) |
| `auth.oidc.provider-type` | `generic` | Preset key from the table above |
| `auth.oidc.allowed-audiences` | \-  | List of acceptable `aud` claims |
| `auth.oidc.verify-expiration` | `true` | Enforce `exp` claim |
| `auth.oidc.clock-skew-seconds` | `300` | Allowed `exp`/`nbf` drift |
| `auth.oidc.username-claim` | `sub` | Claim mapped to `auth.username` |
| `auth.oidc.email-claim` | `email` | Claim mapped to `auth.email` |
| `auth.oidc.roles-claim` | `roles` | Flat roles claim |
| `auth.oidc.role-claim-path` | \-  | Dotted path for nested roles (e.g. `realm_access.roles`) |
| `auth.oidc.groups-claim` | `groups` | Claim mapped to `auth.groups` |
| `auth.oidc.scopes` | preset-specific | OAuth scopes requested in token exchange |
| `auth.oidc.jwks-cache-hours` | `24` | JWKS document cache TTL |
| `auth.oidc.enable-client-credentials` | `false` | Allow client-credentials grant exchange |
| `auth.oidc.enable-refresh-tokens` | `false` | Allow refresh-token grant exchange |

### Keycloak Example (nested roles)

Keycloak puts roles under `realm_access.roles`. Use `role-claim-path`:

```
auth:  enabled: true  type: oidc  oidc:    provider-type: keycloak    issuer-url: https://keycloak.example.com/realms/myrealm    client-id: '${KEYCLOAK_CLIENT_ID}'    client-secret: '${KEYCLOAK_CLIENT_SECRET}'    allowed-audiences:      - account    role-claim-path: realm_access.roles    scopes: [openid, profile, email]
```

Sample Keycloak access-token payload (only relevant fields shown):

```
{  "sub": "f:abc:user1",  "iss": "https://keycloak.example.com/realms/myrealm",  "aud": "account",  "preferred_username": "alice",  "email": "alice@example.com",  "realm_access": {    "roles": ["api-user", "report-viewer"]  }}
```

### Microsoft Azure AD Example

```
auth:  enabled: true  type: oidc  oidc:    provider-type: microsoft    issuer-url: https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0    client-id: '${AZURE_CLIENT_ID}'    allowed-audiences:      - api://my-api    roles-claim: roles
```

### Generic OIDC Example

```
auth:  enabled: true  type: oidc  oidc:    provider-type: generic    issuer-url: https://idp.example.com    client-id: my-api    allowed-audiences: [my-api]    scopes: [openid, profile, email]    jwks-cache-hours: 12
```

## AWS Secrets Manager

Use AWS Secrets Manager as a backing store for Basic Auth credentials. flAPI reads the secret, materializes it into a DuckDB table, and authenticates against that table on each request.

```
auth:  enabled: true  type: basic  from-aws-secretmanager:    secret-name: prod/api/credentials    secret-table: api_users    region: us-east-1    secret-id: '${AWS_ACCESS_KEY}'    secret-key: '${AWS_SECRET_KEY}'    init: |      INSTALL aws;      LOAD aws;
```

| Key | Description |
| --- | --- |
| `auth.from-aws-secretmanager.secret-name` | Name of the AWS secret |
| `auth.from-aws-secretmanager.secret-table` | DuckDB table to materialize secret into |
| `auth.from-aws-secretmanager.region` | AWS region |
| `auth.from-aws-secretmanager.secret-id` | AWS access key ID |
| `auth.from-aws-secretmanager.secret-key` | AWS secret access key |
| `auth.from-aws-secretmanager.init` | Optional SQL run at startup |

The DuckDB Secret Manager backs this lookup; you must define a matching DuckDB secret of type `S3` so flAPI can authenticate to AWS.

## MCP Per-Method Auth

The MCP server has its own `mcp.auth.*` block. Authentication can additionally be required or relaxed per MCP method:

```
mcp:  enabled: true  port: 8081  auth:    enabled: true    type: bearer    jwt-secret: '${MCP_JWT_SECRET}'    methods:      tools/list:        required: false      # public method discovery      tools/call:        required: true       # auth enforced for invocations      resources/read:        required: true
```

Each entry under `mcp.auth.methods.<method>.required` overrides the global `mcp.auth.enabled` flag for that specific MCP method.

## Using Auth Context in SQL Templates

Authenticated user data is available to Mustache templates via the `auth` context:

| Variable | Description |
| --- | --- |
| `{{{auth.username}}}` | Authenticated username / subject |
| `{{{auth.email}}}` | Email claim (OIDC) |
| `{{{auth.roles}}}` | Roles array as joined string |
| `{{{auth.type}}}` | Active auth type (`basic`, `bearer`, `oidc`, ...) |

Row-level security example. `auth.roles` is a comma-joined string (e.g. `"read,admin"`), so match it with `LIKE` rather than Mustache section iteration:

```
SELECT order_id, customer_id, total_amountFROM ordersWHERE 1=1  AND (    -- Admins see everything    '{{auth.roles}}' LIKE '%admin%'    -- Everyone else sees only their own orders    OR customer_id = '{{{auth.username}}}'  )ORDER BY order_date DESC
```

## Error Responses

### 401 Unauthorized

Missing or invalid credentials. flAPI sets `WWW-Authenticate: Basic realm="flAPI"`.

```
HTTP/1.1 401 UnauthorizedWWW-Authenticate: Basic realm="flAPI"
```

### 403 Forbidden

Returned by your SQL when role checks reject the user (flAPI does not have a config-driven role gate; enforce roles in SQL or at the gateway).

## Complete Example

```
# sqls/customers/customer-common.yamlauth-prod:  enabled: true  type: bearer  jwt-secret: '{{env.CUSTOMER_API_JWT_SECRET}}'  jwt-issuer: '{{env.CUSTOMER_API_JWT_ISSUER}}'
```

```
# sqls/customers/customers-rest.yamlurl-path: /customers/{{include:request from customer-common.yaml}}{{include:auth-prod from customer-common.yaml}}{{include:rate-limit from customer-common.yaml}}{{include:connection from customer-common.yaml}}{{include:template-source from customer-common.yaml}}with-pagination: true
```

## Security Best Practices

*   Use environment variables for `jwt-secret`, `client-secret`, and AWS keys.
*   Set `enforce-https.enabled: true` in production (`flapi.yaml`).
*   Use `allowed-audiences` on every OIDC config to bind tokens to your API.
*   Keep `jwks-cache-hours` low if your IdP rotates keys frequently.
*   Prefer OIDC over `jwt`/`bearer` for asymmetric signature validation.
*   Never embed plaintext passwords in committed YAML; MD5 hashing is supported by `verifyPassword` for Basic Auth but is not strong by modern standards.

## Next Steps

*   **[Validation](/docs/endpoints/validation.md)**: Validate and sanitize inputs
*   **[Response Format](/docs/endpoints/response-format.md)**: Structure API responses
*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: DuckLake-backed caching

([🍪 Cookie Settings](#cookie-settings))

# Endpoints Overview

Endpoints are the heart of flAPI. Each YAML file in the `template.path` directory defines exactly one endpoint — a **REST** endpoint, an **MCP tool**, an **MCP resource**, or an **MCP prompt**. This guide covers REST endpoints; see ([MCP](#mcp-endpoints)) below for the MCP variants.

## What is an Endpoint?

A REST endpoint combines:

1.  **URL path** — where clients reach the API (e.g., `/customers/`, `/customers/:id`).
2.  **SQL template** — the Mustache-templated query that runs against your connection.
3.  **Configuration** — request parameters, validation, caching, auth, rate limits.

## File Structure

Every endpoint typically pairs a YAML file with a SQL file:

```
project/├── flapi.yaml           # Main configuration└── sqls/    ├── customers.yaml   # Endpoint configuration    ├── customers.sql    # SQL template    ├── orders.yaml    └── orders.sql
```

## Minimal Endpoint

**`sqls/hello.yaml`:**

```
url-path: /hello/method: GETtemplate-source: hello.sqlconnection:  - my-database
```

**`sqls/hello.sql`:**

```
SELECT 'Hello, World!' AS message
```

Start flAPI and `GET /hello/` returns the row.

## REST Endpoint Configuration

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `url-path` | string | required | HTTP path, e.g. `/customers/:id` |
| `method` | string | `GET` | One of `GET`, `POST`, `PUT`, `PATCH`, `DELETE` |
| `template-source` | string | required | Path to a `.sql` Mustache template |
| `connection` | list\[string\] | required | Connection name(s) from `flapi.yaml` |
| `with-pagination` | bool | `true` | Wrap responses in `{data, next, total_count}` and accept `limit`/`offset` |
| `request-fields-validation` | bool | `false` | Reject requests containing parameters not declared under `request:` |
| `request` | list | –   | Request parameters — see [Parameters](/docs/endpoints/parameters.md) |
| `auth` | mapping | –   | Per-endpoint authentication — see [Authentication](/docs/endpoints/authentication.md) |
| `rate-limit` | mapping | –   | Per-endpoint rate limit |
| `cache` | mapping | –   | DuckLake caching config |
| `operation` | mapping | auto | Override read/write detection — see [Write Operations](/docs/endpoints/write-operations.md) |
| `heartbeat` | mapping | –   | Endpoint heartbeat / health-check params |

### Complete REST Endpoint

```
url-path: /customers/:idmethod: GETrequest:  - field-name: id    field-in: path    description: Customer ID    required: true    validators:      - type: int        min: 1  - field-name: segment    field-in: query    description: Market segment    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY]template-source: customers.sqlconnection:  - customers-parquetwith-pagination: truerequest-fields-validation: falseauth:  enabled: true  type: basic  users:    - username: admin      password: '${ADMIN_PASSWORD}'      roles: [admin, read]rate-limit:  enabled: true  max: 1000  interval: 60   # secondscache:  enabled: true  table: customers_cache  schema: analytics  schedule: 5m  primary-key: [id]  cursor:    column: registration_date    type: date
```

## URL Paths

### Basic Paths

```
url-path: /customers/url-path: /products/search/url-path: /analytics/revenue/
```

Rules:

*   Must start with `/`.
*   Lowercase with hyphens (`/product-categories/`).
*   Trailing slash is optional but conventional.

### Path Parameters

flAPI uses **colon prefixes**, not curly braces:

```
url-path: /customers/:id
```

```
request:  - field-name: id    field-in: path    required: true    validators:      - type: int        min: 1
```

Examples:

*   `/customers/:id` → `params.id`
*   `/orders/:order_id/items/:item_id` → `params.order_id`, `params.item_id`

## Request Parameters

```
request:  - field-name: segment    field-in: query    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING]  - field-name: id    field-in: path    required: true    validators:      - type: int        min: 1  - field-name: x_region    field-in: header    validators:      - type: enum        allowedValues: [US, EU, APAC]  - field-name: name    field-in: body    required: true    validators:      - type: string        min: 2        max: 100
```

See [Parameters](/docs/endpoints/parameters.md) for the four locations and [Validation](/docs/endpoints/validation.md) for the seven validator types.

## SQL Templates

```
template-source: customers.sql
```

```
-- sqls/customers.sqlSELECT  c_custkey    AS id,  c_name       AS name,  c_mktsegment AS segmentFROM customersWHERE 1=1{{#params.segment}}  AND c_mktsegment = '{{{ params.segment }}}'{{/params.segment}}{{#params.id}}  AND c_custkey = {{ params.id }}{{/params.id}}
```

See [SQL Templating](/docs/concepts/sql-templating.md) for the full guide.

## Data Connections

### Single Connection

```
connection:  - customers-parquet
```

### Multiple Connections

flAPI attaches every listed connection to the DuckDB session, so you can join across sources:

```
connection:  - bigquery-warehouse  - customers-parquet
```

The first connection's properties are exposed as `conn.*` in the template:

```
SELECT *FROM read_parquet('{{{ conn.path }}}')
```

## Response Format

By default flAPI returns JSON wrapped in an envelope:

```
{  "data": [    { "id": 1, "name": "Customer A" },    { "id": 2, "name": "Customer B" }  ],  "next": "/customers/?offset=100&limit=100",  "total_count": 1532}
```

Disable pagination to return the raw array, or use `?format=csv` / `?format=arrow` for other formats. See [Response Format](/docs/endpoints/response-format.md).

## MCP Endpoints

flAPI exposes the same YAML configuration as three MCP shapes for AI agents. Each MCP endpoint type uses a different top-level key.

### MCP Tool

A callable tool that runs an SQL query.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `mcp-tool.name` | string | required | Unique tool name (alphanumeric / underscore) |
| `mcp-tool.description` | string | required | Tool description for the LLM |
| `mcp-tool.result-mime-type` | string | `application/json` | MIME type returned to the model |

```
mcp-tool:  name: customer_lookup  description: Retrieve customer information by ID or filter criteria  result-mime-type: application/jsonrequest:  - field-name: id    field-in: query    required: false    validators:      - type: int        min: 1template-source: customers.sqlconnection:  - customers-parquet
```

Only `name`, `description`, and `result-mime-type` are recognised under `mcp-tool:`. Any other keys are ignored.

### MCP Resource

A readable resource (e.g. a schema dump).

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `mcp-resource.name` | string | required | Unique resource name |
| `mcp-resource.description` | string | required | Resource description |
| `mcp-resource.mime-type` | string | `application/json` | Content MIME type |

```
mcp-resource:  name: customer_schema  description: Customer database schema and field definitions  mime-type: application/jsontemplate-source: schema-query.sqlconnection:  - customers-db
```

### MCP Prompt

A reusable, parameterised Mustache prompt. Prompts do **not** require `connection` or `template-source` — the prompt body is inline.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `mcp-prompt.name` | string | required | Unique prompt name |
| `mcp-prompt.description` | string | required | Prompt description |
| `mcp-prompt.template` | string | required | Inline Mustache template |
| `mcp-prompt.arguments` | list\[string\] | `[]` | Template argument names |

```
mcp-prompt:  name: customer_analysis  description: Generate a customer analysis prompt  template: |    You are a data analyst. Analyse this customer:    {{#customer_id}}Customer ID: {{customer_id}}{{/customer_id}}    {{#segment}}Segment: {{segment}}{{/segment}}    Provide insights on purchasing patterns and recommendations.  arguments:    - customer_id    - segment
```

## Endpoint Discovery

```
$ flapii endpoints listAvailable EndpointsPath                  Method  Cached/customers/           GET     Yes/customers/:id        GET     Yes/orders/              POST    No/analytics/revenue/   GET     Yes
```

```
$ flapii endpoints describe /customers/Endpoint:   /customers/Method:     GETTemplate:   sqls/customers.sqlConnection: customers-parquetCache:      Enabled (5m schedule)Auth:       Required (basic)Rate Limit: 1000 req/min
```

## OpenAPI Documentation

flAPI auto-generates an OpenAPI document at:

```
GET /docs
```

Each `description:` and validator becomes part of the spec.

## Common Patterns

### Pagination

`limit` and `offset` are always accepted when `with-pagination: true`; declare them only when you want to constrain them.

```
with-pagination: truerequest:  - field-name: limit    field-in: query    validators:      - type: int        min: 1        max: 1000  - field-name: offset    field-in: query    validators:      - type: int        min: 0
```

### Search

```
request:  - field-name: search    field-in: query    validators:      - type: string        min: 3        max: 100        regex: '^[A-Za-z0-9 _-]+$'
```

```
SELECT * FROM productsWHERE name ILIKE '%{{{ params.search }}}%'
```

### Date Ranges

```
request:  - field-name: start_date    field-in: query    validators:      - type: date  - field-name: end_date    field-in: query    validators:      - type: date
```

```
SELECT * FROM ordersWHERE order_date BETWEEN '{{{ params.start_date }}}' AND '{{{ params.end_date }}}'
```

## Best Practices

1.  **Descriptive URL paths.** `/customers/by-segment/`, not `/api1/`.
2.  **Document every parameter** with `description:` — it shows up in OpenAPI.
3.  **Validate every input.** See [Validation](/docs/endpoints/validation.md).
4.  **Cache read-heavy endpoints** with DuckLake.
5.  **Rate-limit per endpoint** for spiky traffic.
6.  **One YAML per endpoint** — keep them small and focused.

## Troubleshooting

### 404 Endpoint not found

*   File exists in `sqls/`?
*   `url-path` matches request (slashes, colon-prefixes correct)?
*   Server reloaded since the change?

### Template error

*   `template-source` resolves to a real file?
*   Mustache braces balanced?
*   Run `flapii templates expand /endpoint/`.

### Connection error

*   Connection name matches `flapi.yaml`?
*   Connection initialised at server start?

## Next Steps

*   [YAML Syntax](/docs/endpoints/yaml-syntax.md) — extended YAML, includes, env vars
*   [Parameters](/docs/endpoints/parameters.md) — query/path/header/body
*   [Validation](/docs/endpoints/validation.md) — the seven validator types
*   [SQL Templating](/docs/concepts/sql-templating.md) — Mustache patterns
*   [Response Format](/docs/endpoints/response-format.md) — pagination, CSV, Arrow
*   [Authentication](/docs/endpoints/authentication.md) — basic / JWT / bearer / OIDC
*   [Caching](/docs/guides/caching/setup.md) — DuckLake caching

([🍪 Cookie Settings](#cookie-settings))

# Parameters

Parameters let clients customise API requests. flAPI extracts them from four locations — query, path, header, and body — and exposes them to SQL templates as `params.<name>`.

## Parameter Sources

Each `request:` entry uses these top-level keys:

| Key | Required | Description |
| --- | --- | --- |
| `field-name` | yes | Name used in SQL as `params.<field-name>` |
| `field-in` | yes | `query`, `path`, `header`, or `body` |
| `description` | no  | Human-readable description (also surfaces in OpenAPI) |
| `required` | no  | Default `false` |
| `default` | no  | Default value (always a string in YAML) |
| `validators` | no  | List of validators — see [Validation](/docs/endpoints/validation.md) |

## Query Parameters

The most common type — used for filtering and search.

```
# GET /customers?segment=AUTOMOBILE&id=42url-path: /customers/method: GETrequest:  - field-name: segment    field-in: query    description: Market segment to filter by    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY]  - field-name: id    field-in: query    description: Customer ID    required: false    validators:      - type: int        min: 1
```

### Usage in SQL

```
SELECT * FROM customersWHERE 1=1{{#params.segment}}  AND market_segment = '{{{ params.segment }}}'{{/params.segment}}{{#params.id}}  AND id = {{ params.id }}{{/params.id}}
```

### Calls

```
curl http://localhost:8080/customers/curl 'http://localhost:8080/customers/?segment=AUTOMOBILE'curl 'http://localhost:8080/customers/?segment=AUTOMOBILE&id=42'
```

## Path Parameters

Used for resource identification. flAPI uses **colon-prefixed** placeholders in `url-path` (`:id`, `:customer_uuid`, ...).

```
# /customers/:idurl-path: /customers/:idmethod: GETrequest:  - field-name: id    field-in: path    description: Customer ID    required: true    validators:      - type: int        min: 1
```

### SQL Template

```
SELECT  c_custkey  AS id,  c_name     AS name,  c_acctbal  AS balanceFROM customersWHERE c_custkey = {{ params.id }}
```

### Call

```
curl http://localhost:8080/customers/12345
```

### Multiple Path Parameters

```
url-path: /orders/:order_id/items/:item_idmethod: GETrequest:  - field-name: order_id    field-in: path    required: true    validators:      - type: int        min: 1  - field-name: item_id    field-in: path    required: true    validators:      - type: int        min: 1
```

## Header Parameters

For authentication, multi-tenancy, and request metadata.

```
request:  - field-name: api_key    field-in: header    description: API authentication key    required: true    validators:      - type: string        min: 16        max: 64        regex: '^[A-Za-z0-9_-]+$'  - field-name: x_user_region    field-in: header    description: User's region for data filtering    required: false    validators:      - type: enum        allowedValues: [US, EU, APAC, LATAM]
```

```
curl http://localhost:8080/customers/ \  -H 'X-API-Key: your-api-key' \  -H 'X-User-Region: US'
```

## Body Parameters (POST/PUT/PATCH)

When `method:` is `POST`, `PUT`, `PATCH` or `DELETE`, body fields are declared with `field-in: body`. Each field is a top-level key in the JSON body — flAPI does not support nested object/array validators; declare each leaf field separately.

```
url-path: /customers/method: POSTrequest:  - field-name: name    field-in: body    required: true    validators:      - type: string        min: 2        max: 100        regex: '^[A-Za-z ]+$'  - field-name: email    field-in: body    required: true    validators:      - type: email  - field-name: segment    field-in: body    required: true    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE]template-source: customers-create.sqlconnection: [customers-parquet]
```

```
curl -X POST http://localhost:8080/customers/ \  -H 'Content-Type: application/json' \  -d '{    "name": "New Customer",    "email": "customer@example.com",    "segment": "AUTOMOBILE"  }'
```

### SQL Template

```
INSERT INTO customers (name, email, segment)VALUES (  '{{{ params.name }}}',  '{{{ params.email }}}',  '{{{ params.segment }}}')RETURNING *
```

See [Write Operations](/docs/endpoints/write-operations.md) for the full CRUD configuration.

## Default Values

`default:` is always a string in YAML. Validators run after the default is applied, so the default must be valid:

```
- field-name: status  field-in: query  default: "active"  validators:    - type: enum      allowedValues: [active, inactive, pending]- field-name: limit  field-in: query  default: "100"  validators:    - type: int      min: 1      max: 1000
```

In SQL templates, you can also fall back via Mustache:

```
LIMIT {{#params.limit}}{{ params.limit }}{{/params.limit}}{{^params.limit}}100{{/params.limit}}
```

## Common Patterns

### Pagination

Pagination is automatic when `with-pagination` is `true` (the default). The `limit` / `offset` query parameters are always accepted; declare them explicitly only when you want validators on them.

```
with-pagination: true   # defaultrequest:  - field-name: limit    field-in: query    required: false    validators:      - type: int        min: 1        max: 1000  - field-name: offset    field-in: query    required: false    validators:      - type: int        min: 0
```

### Search

```
request:  - field-name: search    field-in: query    required: false    validators:      - type: string        min: 3        max: 100        regex: '^[A-Za-z0-9 _-]+$'
```

```
SELECT * FROM productsWHERE 1=1{{#params.search}}  AND (name ILIKE '%{{{ params.search }}}%' OR description ILIKE '%{{{ params.search }}}%'){{/params.search}}
```

### Sorting

```
request:  - field-name: sort_by    field-in: query    required: false    validators:      - type: enum        allowedValues: [name, price, created_at, popularity]  - field-name: sort_order    field-in: query    required: false    default: "DESC"    validators:      - type: enum        allowedValues: [ASC, DESC]
```

```
SELECT * FROM productsORDER BY {{#params.sort_by}}{{{ params.sort_by }}}{{/params.sort_by}}{{^params.sort_by}}created_at{{/params.sort_by}} {{{ params.sort_order }}}
```

### Date Range

```
request:  - field-name: start_date    field-in: query    required: false    validators:      - type: date        min: "2000-01-01"  - field-name: end_date    field-in: query    required: false    validators:      - type: date
```

```
SELECT * FROM ordersWHERE 1=1{{#params.start_date}}  AND order_date >= '{{{ params.start_date }}}'{{/params.start_date}}{{#params.end_date}}  AND order_date <= '{{{ params.end_date }}}'{{/params.end_date}}
```

## Accessing Parameters in SQL

```
-- Query parameter ?customer_id=123WHERE customer_id = {{ params.customer_id }}-- Path parameter /customers/:idWHERE customer_id = {{ params.id }}-- Header X-Region: USWHERE region = '{{{ params.x_region }}}'-- Body field "name"INSERT INTO customers (name) VALUES ('{{{ params.name }}}')
```

Pick the brace form by validator type:

*   `int`, `date`, `time` (numeric/identifier-like): `{{ params.x }}` (double).
*   `string`, `enum`, `email`, `uuid` (string-like): `'{{{ params.x }}}'` (triple, quoted).

### Conditional Blocks

```
{{#params.status}}  AND status = '{{{ params.status }}}'{{/params.status}}{{^params.include_inactive}}  AND active = true{{/params.include_inactive}}
```

## Best Practices

### 1\. Always validate

```
validators:  - type: enum    allowedValues: [active, inactive]
```

### 2\. Use snake\_case for field names

```
field-name: customer_idfield-name: order_datefield-name: product_category
```

### 3\. Document with `description:`

`description` ends up in the auto-generated OpenAPI doc.

```
- field-name: date_from  field-in: query  description: |    Start date for filtering orders (YYYY-MM-DD).    Defaults to 30 days ago if not provided.  default: "2024-01-01"  validators:    - type: date
```

### 4\. Bound numeric ranges

```
validators:  - type: int    min: 1    max: 1000
```

### 5\. Enable strict field validation for sensitive endpoints

```
request-fields-validation: true   # rejects requests with extra params
```

## Common Issues

### Missing required parameter

```
{  "errors": [{ "field": "customer_id", "message": "Required field is missing" }]}
```

### Invalid value

```
{  "errors": [{ "field": "status", "message": "Invalid enum value" }]}
```

### Parameter not visible in SQL

```
-- Wrong: missing prefixWHERE customer_id = {{ customer_id }}-- CorrectWHERE customer_id = {{ params.customer_id }}
```

## Next Steps

*   [Validation](/docs/endpoints/validation.md) — the seven validator types in detail
*   [SQL Templating](/docs/concepts/sql-templating.md) — Mustache syntax for templates
*   [Endpoints Overview](/docs/endpoints/overview.md) — full endpoint tutorial
*   [Authentication](/docs/endpoints/authentication.md) — secure parameter access

([🍪 Cookie Settings](#cookie-settings))

# Response Format

flAPI returns **JSON by default** and also supports **CSV** and **Apache Arrow IPC stream** via content negotiation. Pagination is controlled per-endpoint with the `with-pagination` flag.

There is no `response:` block in endpoint YAML — formats are negotiated at request time and pagination is configured by a single top-level boolean.

## Default JSON Response

With pagination enabled (the default), the response is:

```
{  "data": [    { "id": 1, "name": "Customer A" },    { "id": 2, "name": "Customer B" }  ],  "next": "/customers/?offset=100&limit=100",  "total_count": 1532}
```

| Field | Type | Description |
| --- | --- | --- |
| `data` | array | Query result rows |
| `next` | string | Path + querystring for the next page, or `""` if no more |
| `total_count` | integer | Total rows matching the query (across all pages) |

With pagination disabled the body is the bare `data` array (no envelope).

In addition to the body, flAPI always sets the following response headers when pagination is on:

| Header | Value |
| --- | --- |
| `X-Total-Count` | Same as `total_count` |
| `X-Offset` | Current offset |
| `X-Limit` | Current limit |
| `X-Next` | Same as `next` |

## Disabling Pagination

```
url-path: /customers/lookup/method: GETtemplate-source: lookup.sqlconnection: [customers-parquet]with-pagination: false   # body is just the rows, no envelope
```

When `with-pagination: false`:

*   The body is the raw `data` array (no `next`/`total_count` envelope).
*   The `limit` and `offset` query parameters are still accepted but no envelope/headers are added.

When `with-pagination: true` (default), the client can send `?limit=N&offset=M`:

```
curl 'http://localhost:8080/customers/?limit=20&offset=40'
```

## Content Negotiation

Clients request a format with either the `Accept` header or the `format` query parameter. The query parameter wins when both are sent.

### JSON (Default)

Content-Type: `application/json`

```
curl http://localhost:8080/customers/curl -H 'Accept: application/json' http://localhost:8080/customers/
```

### CSV

Content-Type: `text/csv`

```
curl 'http://localhost:8080/customers/?format=csv'curl -H 'Accept: text/csv' http://localhost:8080/customers/
```

### Apache Arrow IPC Stream

Content-Type: `application/vnd.apache.arrow.stream`

```
curl 'http://localhost:8080/customers/?format=arrow' -o customers.arrowcurl -H 'Accept: application/vnd.apache.arrow.stream' \     http://localhost:8080/customers/ -o customers.arrow
```

Arrow responses use Arrow's native framing; flAPI disables HTTP compression for them (Arrow handles compression internally via LZ4/ZSTD) and sets `Content-Length` explicitly.

## Error Responses

Validation failures return HTTP `400` with a JSON body listing each invalid field:

```
{  "errors": [    { "field": "segment", "message": "Invalid enum value" },    { "field": "id",      "message": "Integer is less than the minimum allowed value" }  ]}
```

| Status | Meaning |
| --- | --- |
| `200` | Success |
| `400` | Validation error (invalid or missing parameters) |
| `401` | Missing/invalid authentication |
| `403` | Authenticated but forbidden |
| `404` | No endpoint matches the URL |
| `429` | Rate limit exceeded |
| `500` | Query execution or server error |

## Write Endpoints

For `POST`/`PUT`/`PATCH`/`DELETE` endpoints the response shape is different — see [Write Operations](/docs/endpoints/write-operations.md):

```
{  "rows_affected": 1,  "data": [    { "id": 42, "name": "New Widget", "price": 29.99 }  ]}
```

`data` is only present when the endpoint has `operation.returns-data: true` and the SQL template uses a `RETURNING` clause.

## Column Formatting

DuckDB drives the output. Dates and timestamps come back as ISO 8601 strings; numerics keep their precision. Reshape values in SQL when you need a different form:

```
SELECT  STRFTIME(created_at, '%Y-%m-%d') AS date,  ROUND(price, 2)                  AS price,  COALESCE(optional_field, 'N/A')  AS optional_fieldFROM products
```

## Best Practices

*   Leave `with-pagination` at the default unless you have a specific reason — it keeps responses bounded and surfaces total counts.
*   Validate every parameter so 400s come back with clear field-level messages.
*   For bulk exports, prefer `?format=arrow` or `?format=csv` over JSON.

## Next Steps

*   [Parameters](/docs/endpoints/parameters.md) — configure request parameters
*   [Validation](/docs/endpoints/validation.md) — surface clear 400 responses
*   [Write Operations](/docs/endpoints/write-operations.md) — `rows_affected` / `RETURNING`
*   [Caching](/docs/guides/caching/setup.md) — reduce latency on large queries

([🍪 Cookie Settings](#cookie-settings))

# SQL Templates

SQL templates define the queries that power your API endpoints. flAPI uses Mustache templating syntax to create dynamic, parameterized queries.

## Quick Example

```
-- sqls/customers.sqlSELECT  customer_id,  name,  email,  segmentFROM customersWHERE 1=1{{#params.segment}}  AND segment = '{{{ params.segment }}}'{{/params.segment}}{{#params.id}}  AND customer_id = {{ params.id }}{{/params.id}}ORDER BY created_at DESCLIMIT 100
```

## Mustache Syntax

flAPI uses Mustache for template logic:

*   **`{{ params.X }}`** on a **typed field** (validator type `int`/`double`/`boolean`/`date`/`time`/`uuid`/`enum`/`email`/`string`): the renderer emits a DuckDB `?` placeholder and binds the value via `duckdb_bind_*`. SQL injection is structurally impossible at this site. Use for equality lookups.
*   **`{{{ variable }}}`**: Renders the raw value (no escaping). Use when the value participates in the SQL **text** — `LIKE '%...%'`, identifier insertion, or composite expressions the prepared-statement rewriter cannot decompose.
*   **`{{#variable}}...{{/variable}}`**: Conditional block (rendered if variable is present and truthy). References inside a section stay on the Mustache path even if the field is typed.
*   **`{{^variable}}...{{/variable}}`**: Inverted block (rendered if variable is absent or falsy).

> See **[SQL Templating Guide → Variable Insertion](/docs/concepts/sql-templating.md)** for the full layered injection-defense story (validators → prepared bind → regex fallback).

## Available Context

In your SQL templates, you have access to:

*   **`params.*`**: Validated request parameters (query, path, header, body)
*   **`conn.*`**: Properties of the first connection listed in `connection:`
*   **`auth.*`**: Authenticated user info — `auth.username`, `auth.roles`, `auth.email`, `auth.type`, `auth.authenticated`
*   **`cache.*`**: Cache metadata (used in cache-refresh templates)
*   **`env.*`**: Whitelisted environment variables

## Next Steps

For comprehensive SQL templating documentation, see:

*   **[SQL Templating Guide](/docs/concepts/sql-templating.md)**: Complete guide with examples
*   **[Parameters](/docs/endpoints/parameters.md)**: Define request parameters
*   **[Validation](/docs/endpoints/validation.md)**: Validate input safely
*   **[Endpoints Overview](/docs/endpoints/overview.md)**: Complete endpoint configuration
*   **[Examples](/docs/examples/parquet-api.md)**: See templates in action

([🍪 Cookie Settings](#cookie-settings))

# Validation

Parameter validation is critical for security, data integrity, and API reliability. flAPI ships with **seven** built-in validator types — `int`, `string`, `enum`, `email`, `uuid`, `date`, and `time`.

When a field carries a typed validator and is referenced via double-brace `{{ params.X }}` at top level of the template, flAPI **rewrites the reference to a DuckDB `?` placeholder and binds the value via `duckdb_bind_*`** — the value travels as a primitive, never as SQL text. This is the hard boundary; the legacy keyword regex is a fallback for non-bindable sites (triple-brace `{{{ }}}` and untyped fields).

## Why Validation Matters

**Without validation:** unclear 400s, weak typing, regex relies on keyword heuristics. **With validation:** strict type parsing (`1; DROP TABLE` cannot slip through as `1`), DuckDB prepared-statement binding for typed fields, clear error messages, automatic OpenAPI docs.

## Basic Validation

Add one or more validators to any request field:

```
request:  - field-name: customer_id    field-in: query    required: true    validators:      - type: int        min: 1        max: 999999        preventSqlInjection: true
```

`preventSqlInjection` defaults to `true`. For **numeric/temporal bindable types** (`int`, `double`, `boolean`, `date`, `time`) the SQL-keyword regex is now demoted to a debug log — the prepared-statement bind is the real defense, and the regex's notorious false positives (e.g. `latitude=1.111`) are gone. **Varchar-bindable types** (`string`, `uuid`, `email`, `enum`) keep the regex because flAPI templates routinely embed them via triple-brace `{{{ }}}` for `LIKE` patterns. Set `preventSqlInjection: false` only when you have a deliberate reason (e.g. a field that holds an opaque token).

## Validator Types

flAPI supports exactly seven validator types. Anything else is silently ignored by `src/request_validator.cpp`.

### Integer (`int`)

For whole numbers. Range bounds default to `INT_MIN`/`INT_MAX`.

```
validators:  - type: int    min: 1    max: 1000    preventSqlInjection: true
```

### String (`string`)

Length is controlled with `min` and `max` (in characters), and pattern matching uses `regex` (full match required).

```
validators:  - type: string    min: 3    max: 50    regex: '^[A-Za-z0-9_-]+$'    preventSqlInjection: true
```

Use a separate `enum` validator if the field should be one of a fixed set of values — `string` does **not** accept an `enum:` key.

### Enum (`enum`)

Whitelist of allowed values.

```
validators:  - type: enum    allowedValues: [draft, published, archived]
```

### Email (`email`)

Built-in regex: `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`.

```
validators:  - type: email
```

### UUID (`uuid`)

Standard 36-character form (`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`).

```
validators:  - type: uuid
```

### Date (`date`)

Format `YYYY-MM-DD`. Optional `min` / `max` are strings in the same format.

```
validators:  - type: date    min: "2000-01-01"    max: "2025-12-31"
```

### Time (`time`)

Format `HH:MM:SS`. Optional `min` / `max` are strings in the same format.

```
validators:  - type: time    min: "08:00:00"    max: "18:00:00"
```

## SQL Injection Prevention

When `preventSqlInjection` is `true` (the default), flAPI rejects values that contain SQL keywords (`SELECT`, `INSERT`, `UPDATE`, `DELETE`, `DROP`, `UNION`, ...), comment markers (`--`, `/*`, `*/`), or common injection patterns (`' OR 1=1`, `'; ...`).

Only set it to `false` for fields where you intentionally accept SQL-looking content (very rare).

## Multiple Validators on One Field

Stack validators when you need multiple checks:

```
- field-name: username  field-in: query  required: true  validators:    - type: string      min: 3      max: 20      regex: '^[a-zA-Z0-9_]+$'
```

All declared validators must pass for the parameter to be accepted.

## Common Validation Patterns

### Email Address

```
- field-name: email  validators:    - type: email
```

### Phone Number (E.164)

```
- field-name: phone  validators:    - type: string      regex: '^\+?[1-9]\d{1,14}$'
```

### Country Code

```
- field-name: country  validators:    - type: string      regex: '^[A-Z]{2}$'
```

### Status Enum

```
- field-name: status  validators:    - type: enum      allowedValues: [active, inactive, pending]
```

### Customer UUID

```
- field-name: customer_id  validators:    - type: uuid
```

### Date Range

```
- field-name: registration_date  validators:    - type: date      min: "2000-01-01"      max: "2025-12-31"
```

## Strict Mode: Reject Unknown Parameters

By default flAPI accepts any extra query parameter and silently ignores it. Set `request-fields-validation: true` on the endpoint to reject requests that include parameters not declared under `request:`.

```
url-path: /customers/method: GETrequest-fields-validation: true   # default: falserequest:  - field-name: id    field-in: query    validators:      - type: int        min: 1
```

A request containing `?id=1&unknown=foo` returns a `400` with the offending field name. The pagination parameters `offset` and `limit` are always allowed.

## Error Responses

Validation errors come back as HTTP `400` with one entry per failed field:

```
{  "errors": [    { "field": "segment", "message": "Invalid enum value" },    { "field": "id",      "message": "Integer is less than the minimum allowed value" }  ]}
```

The exact messages come from `src/request_validator.cpp`:

| Validator | Message |
| --- | --- |
| `int` (out of range) | `Integer is less than/greater than the minimum/maximum allowed value` |
| `int` (non-numeric) | `Invalid integer value` |
| `string` (too short/long) | `String is shorter/longer than the minimum/maximum allowed length` |
| `string` (regex mismatch) | `Invalid string format` |
| `enum` | `Invalid enum value` |
| `email` | `Invalid email format` |
| `uuid` | `Invalid UUID format` |
| `date` / `time` | `Invalid date/time format`, `Date/Time is before/after the minimum/maximum allowed date/time` |
| any (SQL pattern) | `Potential SQL injection detected` |
| missing required | `Required field is missing` |
| unknown (strict mode) | `Unknown parameter not defined in endpoint configuration` |

## Security Best Practices

### 1\. Always validate user input

```
validators:  - type: enum    allowedValues: [active, inactive]
```

### 2\. Whitelist with `enum` where possible

```
validators:  - type: enum    allowedValues: [name, email, created_at]
```

### 3\. Limit string length and tighten regex

```
validators:  - type: string    min: 1    max: 100    regex: '^[A-Za-z0-9_ \-]+$'
```

### 4\. Restrict numeric ranges

```
validators:  - type: int    min: 1    max: 1000
```

### 5\. Enable strict field mode for sensitive endpoints

```
request-fields-validation: true
```

## Real-World Examples

### Product Search

```
url-path: /products/search/method: GETrequest:  - field-name: query    field-in: query    required: true    validators:      - type: string        min: 3        max: 100  - field-name: category    field-in: query    required: false    validators:      - type: enum        allowedValues: [electronics, clothing, books, home]  - field-name: min_price    field-in: query    required: false    validators:      - type: int        min: 0        max: 1000000  - field-name: limit    field-in: query    required: false    validators:      - type: int        min: 1        max: 100
```

### Date Range Analytics

```
url-path: /analytics/revenue/method: GETrequest:  - field-name: start_date    field-in: query    required: true    validators:      - type: date        min: "2020-01-01"  - field-name: end_date    field-in: query    required: true    validators:      - type: date  - field-name: granularity    field-in: query    required: false    default: day    validators:      - type: enum        allowedValues: [day, week, month, quarter, year]
```

### Customer Lookup by UUID

```
url-path: /customers/:customer_uuidmethod: GETrequest:  - field-name: customer_uuid    field-in: path    required: true    validators:      - type: uuid
```

## Common Issues

### Validators not firing

*   Check indentation under `validators:` — it must be a list of mappings, not a single mapping.
*   Make sure the type is one of the seven supported names (`int`, `string`, `enum`, `email`, `uuid`, `date`, `time`).

### Regex rejects valid input

*   The pattern must match the **whole** value (anchored implicitly).
*   Escape backslashes once for YAML, then once more if needed for the regex engine.

### "Potential SQL injection detected" on a clean value

*   The default scanner blocks SQL keywords as whole words and certain patterns. If the value is legitimate (e.g. a column name in a CMS search), set `preventSqlInjection: false` on that validator and rely on a regex / enum instead.

## Next Steps

*   [Parameters](/docs/endpoints/parameters.md) — learn the four parameter sources
*   [SQL Templating](/docs/concepts/sql-templating.md) — use validated params safely
*   [Endpoints Overview](/docs/endpoints/overview.md) — full endpoint tutorial
*   [Authentication](/docs/endpoints/authentication.md) — secure your endpoints

([🍪 Cookie Settings](#cookie-settings))

# Write Operations (CRUD)

flAPI supports full **CRUD** (Create, Read, Update, Delete) operations, so you can build complete data APIs that not only serve data but also accept modifications. This guide covers how to configure write endpoints with validation, transactions, and `RETURNING`\-clause responses.

## Overview

Write operations in flAPI let you:

*   **CREATE** (POST) — insert new records with validation
*   **UPDATE** (PUT / PATCH) — modify existing records (full or partial)
*   **DELETE** (DELETE) — remove records safely
*   **RETURNING** — get the affected rows back in the response

All write endpoints support:

*   Pre-write validation (the same 7 validator types as read endpoints + SQL-injection scanning)
*   ACID transactions (automatic rollback on error)
*   `RETURNING`\-clause data in the response
*   Structured `400` error responses with field-level messages

## How flAPI Picks Read vs Write

flAPI auto-detects the operation type from `method:`:

| `method:` | `operation.type` default |
| --- | --- |
| `GET` | `Read` |
| `POST`, `PUT`, `PATCH`, `DELETE` | `Write` |

You can override the auto-detection with an explicit `operation:` block.

## Basic Configuration

Write endpoints look just like read endpoints, with an additional `operation:` block.

```
url-path: /api/products/method: POSToperation:  type: Write              # Read or Write (case-insensitive)  validate-before-write: true   # default true  returns-data: true       # populate `data` from RETURNING  transaction: true        # wrap in a transaction (default true)request:  - field-name: product_name    field-in: body    required: true    validators:      - type: string        min: 1        max: 100        preventSqlInjection: truetemplate-source: products-create.sqlconnection:  - northwind-sqlite
```

| `operation.*` key | Type | Default | Description |
| --- | --- | --- | --- |
| `type` | string | auto | `Read` or `Write` (case-insensitive). Auto-set from `method:`. |
| `returns-data` | bool | `false` | Populate `data` from a `RETURNING` clause. |
| `transaction` | bool | `true` | Wrap the query in a transaction. |
| `validate-before-write` | bool | `true` | Enforce request validation before any SQL runs. |

## Operation Types

### CREATE (POST)

Insert new records.

```
url-path: /api/products/method: POSToperation:  type: Write  returns-data: true  transaction: true  validate-before-write: truerequest:  - field-name: product_name    field-in: body    required: true    validators:      - type: string        min: 1        max: 100        preventSqlInjection: true  - field-name: supplier_id    field-in: body    required: true    validators:      - type: int        min: 1  - field-name: unit_price    field-in: body    required: false    validators:      - type: string        regex: '^\d+(\.\d{1,2})?$'template-source: products-create.sqlconnection:  - northwind-sqlite
```

**SQL template (`products-create.sql`):**

```
INSERT INTO nw.Products (    ProductName,    SupplierID,    UnitPrice)VALUES (    '{{{ params.product_name }}}',    {{ params.supplier_id }},    {{ params.unit_price }})RETURNING ProductID, ProductName, UnitPrice
```

**Request:**

```
curl -X POST http://localhost:8080/api/products/ \  -H 'Content-Type: application/json' \  -d '{    "product_name": "New Widget",    "supplier_id": 1,    "unit_price": "29.99"  }'
```

**Response:**

```
{  "rows_affected": 1,  "data": [    { "ProductID": 42, "ProductName": "New Widget", "UnitPrice": 29.99 }  ]}
```

### UPDATE (PUT / PATCH)

Modify existing records. For partial updates, leave the body fields optional and `COALESCE` in SQL.

```
url-path: /api/products/:product_idmethod: PUToperation:  type: Write  returns-data: true  transaction: true  validate-before-write: truerequest:  - field-name: product_id    field-in: path    required: true    validators:      - type: int        min: 1  - field-name: product_name    field-in: body    required: false    validators:      - type: string        min: 1        max: 100        preventSqlInjection: true  - field-name: unit_price    field-in: body    required: false    validators:      - type: string        regex: '^\d+(\.\d{1,2})?$'template-source: products-update.sqlconnection:  - northwind-sqlite
```

**SQL template:**

```
UPDATE nw.ProductsSET    ProductName = COALESCE('{{{ params.product_name }}}', ProductName),    UnitPrice   = COALESCE({{ params.unit_price }}, UnitPrice)WHERE ProductID = {{ params.product_id }}RETURNING ProductID, ProductName, UnitPrice
```

```
curl -X PUT http://localhost:8080/api/products/42 \  -H 'Content-Type: application/json' \  -d '{"product_name": "Updated Widget", "unit_price": "34.99"}'
```

### DELETE

Remove records.

```
url-path: /api/products/:product_idmethod: DELETEoperation:  type: Write  returns-data: false  transaction: true  validate-before-write: truerequest:  - field-name: product_id    field-in: path    required: true    validators:      - type: int        min: 1template-source: products-delete.sqlconnection:  - northwind-sqlite
```

**SQL template:**

```
DELETE FROM nw.ProductsWHERE ProductID = {{ params.product_id }}
```

**Response (no RETURNING):**

```
{  "rows_affected": 1}
```

## Validation

Pre-write validation uses the same seven validator types as read endpoints (`int`, `string`, `enum`, `email`, `uuid`, `date`, `time`). See [Validation](/docs/endpoints/validation.md) for the complete reference.

```
- field-name: email  field-in: body  required: true  validators:    - type: email
```

By default every validator scans the value for SQL-injection patterns (`preventSqlInjection: true`). Disable it only for fields that legitimately contain SQL-looking text.

## Transactions

```
operation:  type: Write  transaction: true   # default
```

Benefits:

*   **Atomicity** — the whole template runs in one transaction; an error rolls everything back.
*   **Consistency** — never partial writes.
*   **Isolation** — concurrent writes don't interleave.
*   **Durability** — committed changes persist.

## RETURNING Clause

Use SQL `RETURNING` to send the affected rows back without a follow-up query. Combine it with `operation.returns-data: true`:

```
operation:  type: Write  returns-data: true
```

```
INSERT INTO products (name, price)VALUES ('{{{ params.name }}}', {{ params.price }})RETURNING id, name, price
```

```
{  "rows_affected": 1,  "data": [    { "id": 42, "name": "Widget", "price": 29.99 }  ]}
```

## Cache Hooks on Write

If the endpoint touches a cached table, two flags control what happens to that cache after the write:

```
cache:  enabled: true  table: products_cache  schema: analytics  schedule: 5m  primary-key: [ProductID]  cursor:    column: UpdatedAt    type: timestamp  invalidate-on-write: true    # mark cache stale after this endpoint runs  refresh-on-write: false      # immediately re-materialise the cache after write
```

| Key | Default | Effect |
| --- | --- | --- |
| `cache.invalidate-on-write` | `false` | Mark the cache stale; the next read triggers a refresh. |
| `cache.refresh-on-write` | `false` | Refresh the cache immediately after the write completes. |

## Response Shape

### Successful write

```
{  "rows_affected": 1,  "data": [    { "id": 42, "name": "Widget", "price": 29.99 }  ]}
```

`data` is only present when `operation.returns-data: true` and the SQL has a `RETURNING` clause.

### Validation error (400)

```
{  "errors": [    { "field": "product_name", "message": "String is shorter than the minimum allowed length" },    { "field": "supplier_id",  "message": "Integer is less than the minimum allowed value" }  ]}
```

### Database error (500)

```
{  "error": "Internal Server Error: FOREIGN KEY constraint failed"}
```

## Best Practices

### 1\. Validate every body field

```
- field-name: email  field-in: body  required: true  validators:    - type: email
```

### 2\. Use enums for status / category fields

```
- field-name: status  field-in: body  validators:    - type: enum      allowedValues: [pending, active, archived]
```

### 3\. Keep transactions on

```
operation:  transaction: true
```

### 4\. Return data for confirmation

```
operation:  returns-data: true
```

```
INSERT INTO products (...) VALUES (...)RETURNING id, name, price
```

### 5\. Validate path parameters too

```
- field-name: product_id  field-in: path  required: true  validators:    - type: int      min: 1
```

### 6\. Use COALESCE for partial updates

```
UPDATE productsSET    name  = COALESCE('{{{ params.name }}}', name),    price = COALESCE({{ params.price }}, price)WHERE id = {{ params.id }}
```

## Security Considerations

### Authentication

Add auth to write endpoints. The schema mirrors read endpoints — see [Authentication](/docs/endpoints/authentication.md).

```
auth:  enabled: true  type: bearer  jwt-secret: '${JWT_SECRET}'  jwt-issuer: 'my-auth-server'
```

For basic auth with roles:

```
auth:  enabled: true  type: basic  users:    - username: editor      password: '${EDITOR_PASSWORD}'      roles: [editor, write]    - username: admin      password: '${ADMIN_PASSWORD}'      roles: [admin, editor, write, read]
```

### Rate Limiting

```
rate-limit:  enabled: true  max: 100  interval: 60     # 100 requests per 60 seconds
```

### Row-Level Restrictions

Use the `auth.*` context inside the SQL template to scope writes to the caller:

```
UPDATE productsSET name = '{{{ params.name }}}'WHERE id = {{ params.id }}  AND (    '{{{ auth.roles }}}' LIKE '%admin%'    OR owner_username = '{{{ auth.username }}}'  )
```

The `auth.*` context exposes `auth.username`, `auth.roles`, `auth.email`, `auth.type`, and `auth.authenticated`.

## Examples

See the [CRUD API Tutorial](/docs/examples/crud-api.md) for a complete working example with all four operations.

## Next Steps

*   [Parameters](/docs/endpoints/parameters.md) — `field-in: body` and friends
*   [Validation](/docs/endpoints/validation.md) — the seven validator types
*   [Authentication](/docs/endpoints/authentication.md) — basic / JWT / bearer / OIDC
*   [Response Format](/docs/endpoints/response-format.md) — read-endpoint envelope vs write-endpoint shape

([🍪 Cookie Settings](#cookie-settings))

# YAML Syntax & Structure

flAPI extends standard YAML with **section includes**, **environment-variable substitution**, and a fixed set of top-level keys per endpoint. This guide covers the extended syntax, the supported endpoint shapes (REST + 3 MCP variants), and path resolution rules.

## Extended YAML Features

flAPI's YAML parser supports standard YAML plus these extensions:

### 1\. Section Includes

flAPI's include directive copies one named section out of another YAML file. The syntax is:

```
{{include:section_name from path/to/file.yaml}}
```

No quotes around the path. The path is resolved relative to the including file.

**Includable sections** (anything else is rejected):

*   `request`
*   `auth`
*   `rate-limit`
*   `connection`
*   `template-source`
*   `cache`
*   `heartbeat`

**Example — a shared `customer-common.yaml`:**

```
# common/customer-common.yamlrequest:  - field-name: id    field-in: query    validators:      - type: int        min: 1auth:  enabled: true  type: basic  users:    - username: admin      password: '${ADMIN_PASSWORD}'      roles: [admin, read]rate-limit:  enabled: true  max: 100  interval: 60connection:  - customers-parquettemplate-source: customers.sqlcache:  enabled: true  table: customers_cache  schema: analytics  schedule: 5m
```

**Endpoint that pulls them in:**

```
# customers-rest.yamlurl-path: /customers/method: GET{{include:request       from common/customer-common.yaml}}{{include:auth          from common/customer-common.yaml}}{{include:rate-limit    from common/customer-common.yaml}}{{include:connection    from common/customer-common.yaml}}{{include:template-source from common/customer-common.yaml}}{{include:cache         from common/customer-common.yaml}}with-pagination: true
```

You can also keep multiple variants of a section under different top-level keys in the source file (`auth`, `auth-dev`, `auth-prod`, ...) and choose which to include:

```
{{include:auth-prod from common/customer-common.yaml}}
```

This works because the include directive grabs the value of whatever top-level key you name.

### 2\. Environment Variables

Reference environment variables anywhere a value is expected. Two equivalent forms exist:

```
auth:  jwt-secret: '${JWT_SECRET}'  jwt-issuer: '{{env.JWT_ISSUER}}'
```

*   `${VAR}` — shell-style.
*   `{{env.VAR}}` — Mustache-style, also works inside SQL templates.

Environment variables must be whitelisted in `flapi.yaml` (`template.environment-whitelist`). Unknown / unwhitelisted variables cause a load-time error.

### 3\. Template Variables

These context variables are available inside SQL templates (not in endpoint YAML):

*   `params.*` — validated request parameters
*   `conn.*` — properties of the first connection listed under `connection:`
*   `auth.*` — authenticated user (`auth.username`, `auth.roles`, `auth.email`, `auth.type`, `auth.authenticated`)
*   `cache.*` — cache metadata (in cache-refresh templates)
*   `env.*` — whitelisted environment variables

See [SQL Templating](/docs/concepts/sql-templating.md) for the full reference.

## Endpoint YAML Structure

Each YAML file defines **one** endpoint of one of four kinds. flAPI decides which kind by inspecting the top-level keys:

| Top-level key present | Endpoint kind |
| --- | --- |
| `url-path` | REST |
| `mcp-tool` | MCP tool |
| `mcp-resource` | MCP resource |
| `mcp-prompt` | MCP prompt |

`url-path` can also coexist with `mcp-tool` / `mcp-resource` when the same query should be exposed both ways.

### Basic REST Endpoint

```
# sqls/customers.yamlurl-path: /customers/method: GETrequest:  - field-name: segment    field-in: query    description: Market segment to filter by    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY]  - field-name: min_balance    field-in: query    description: Minimum account balance    required: false    validators:      - type: int        min: 0        max: 1000000template-source: customers.sqlconnection:  - customers-parquetwith-pagination: truecache:  enabled: true  table: customers_cache  schema: analytics  schedule: 5m  primary-key: [id]
```

### Full-Featured REST Endpoint

```
# sqls/orders.yamlurl-path: /orders/:order_idmethod: GETrequest:  - field-name: order_id    field-in: path    description: Order ID    required: true    validators:      - type: int        min: 1  - field-name: status    field-in: query    description: Order status filter    required: false    default: "completed"    validators:      - type: enum        allowedValues: [pending, completed, cancelled, refunded]  - field-name: start_date    field-in: query    description: Start of the date range (YYYY-MM-DD)    required: false    validators:      - type: date        min: "2020-01-01"  - field-name: end_date    field-in: query    description: End of the date range (YYYY-MM-DD)    required: false    validators:      - type: datetemplate-source: orders/detail.sqlconnection:  - bigquery-warehouse  - customers-parquetwith-pagination: truerequest-fields-validation: trueauth:  enabled: true  type: bearer  jwt-secret: '${JWT_SECRET}'  jwt-issuer: 'my-auth-server'rate-limit:  enabled: true  max: 1000  interval: 60cache:  enabled: true  table: orders_cache  schema: analytics  schedule: 15m  primary-key: [order_id]  cursor:    column: updated_at    type: timestamp  retention:    keep-last-snapshots: 5    max-snapshot-age: 14d
```

## Top-Level Keys Reference

| Key | Type | Default | Notes |
| --- | --- | --- | --- |
| `url-path` | string | –   | Required for REST endpoints. Path parameters use `:name`. |
| `method` | string | `GET` | Single HTTP method: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`. Use one string, not a list. |
| `template-source` | string | –   | Required for REST + MCP-tool + MCP-resource. Path to `.sql` file. |
| `connection` | list\[string\] | –   | Required when a template is used. First entry exposes `conn.*` in templates. |
| `request` | list | `[]` | Request parameter definitions. |
| `with-pagination` | bool | `true` | Wraps responses in `{data, next, total_count}` and accepts `limit`/`offset`. |
| `request-fields-validation` | bool | `false` | Reject requests containing parameters not declared in `request:`. |
| `auth` | mapping | inherits global | Per-endpoint auth override — see [Authentication](/docs/endpoints/authentication.md). |
| `rate-limit` | mapping | inherits global | Per-endpoint rate-limit override. |
| `cache` | mapping | –   | DuckLake cache configuration. |
| `operation` | mapping | auto | `type: Read` / `type: Write`, plus `returns-data`, `transaction`, `validate-before-write`. |
| `heartbeat` | mapping | –   | Endpoint heartbeat config. |
| `mcp-tool` | mapping | –   | Declares an MCP tool. |
| `mcp-resource` | mapping | –   | Declares an MCP resource. |
| `mcp-prompt` | mapping | –   | Declares an MCP prompt. |

> Note: `method` is a single string. There is no `methods: [GET]` list form.

## MCP Endpoint Variants

flAPI exposes the same YAML configuration as three MCP shapes for AI agents.

### MCP Tool

A callable tool that runs an SQL query. Only these three keys are recognised:

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `mcp-tool.name` | string | required | Unique tool name (alphanumeric / underscore) |
| `mcp-tool.description` | string | required | Tool description for the LLM |
| `mcp-tool.result-mime-type` | string | `application/json` | MIME type sent back to the model |

```
# sqls/customers-mcp.yamlmcp-tool:  name: search_customers  description: |    Search and retrieve customer information with flexible filtering    by segment, balance, name, or registration date.  result-mime-type: application/jsonrequest:  - field-name: segment    field-in: query    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, MACHINERY, HOUSEHOLD]  - field-name: min_balance    field-in: query    required: false    validators:      - type: int        min: 0        max: 1000000  - field-name: name    field-in: query    required: false    validators:      - type: string        min: 2        max: 50        regex: '^[A-Za-z ]+$'template-source: customers/search.sqlconnection:  - customers-parquet
```

### MCP Resource

A readable resource — e.g. an introspection query exposed as data.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `mcp-resource.name` | string | required | Unique resource name |
| `mcp-resource.description` | string | required | Description shown to the model |
| `mcp-resource.mime-type` | string | `application/json` | Content MIME type |

```
mcp-resource:  name: customer_schema  description: Customer table schema and field definitions  mime-type: application/jsontemplate-source: customers/schema.sqlconnection:  - customers-parquet
```

### MCP Prompt

A reusable, parameterised Mustache prompt. Prompts have an **inline** template (no `template-source`) and do **not** need a `connection`.

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `mcp-prompt.name` | string | required | Unique prompt name |
| `mcp-prompt.description` | string | required | Description shown to the model |
| `mcp-prompt.template` | string | required | Inline Mustache template body |
| `mcp-prompt.arguments` | list\[string\] | `[]` | Template argument names |

```
mcp-prompt:  name: customer_analysis  description: Generate a customer analysis prompt  template: |    You are a data analyst. Analyse this customer:    {{#customer_id}}Customer ID: {{customer_id}}{{/customer_id}}    {{#segment}}Segment: {{segment}}{{/segment}}    Provide insights on purchasing patterns and recommendations.  arguments:    - customer_id    - segment
```

## File Structure Best Practices

### Recommended Directory Layout

```
project/├── flapi.yaml                  # Main configuration├── common/                     # Shared section definitions│   ├── customer-common.yaml│   └── _shared/│       └── auth.yaml│└── sqls/                       # Endpoint definitions + SQL templates    ├── customers/    │   ├── list.yaml           # GET /customers/    │   ├── list.sql    │   ├── detail.yaml         # GET /customers/:id    │   ├── detail.sql    │   └── search.yaml         # MCP search tool    │    ├── orders/    │   ├── list.yaml    │   ├── list.sql    │   ├── create.yaml         # POST /orders/    │   └── create.sql    │    └── analytics/        ├── revenue.yaml        └── revenue.sql
```

### Naming Conventions

```
sqls/customers/list.yaml      -> GET    /customers/sqls/customers/detail.yaml    -> GET    /customers/:idsqls/customers/search.yaml    -> GET    /customers/search/sqls/orders/create.yaml       -> POST   /orders/sqls/analytics/revenue.yaml   -> GET    /analytics/revenue/
```

## Path Resolution

### Includes

Includes resolve relative to the **including** file:

```
# flapi.yaml (project root) — main config does not use the include directive,# it lives only inside endpoint YAML files.# sqls/customers-rest.yaml{{include:request from ../common/customer-common.yaml}}# Resolves to: common/customer-common.yaml
```

### Template Source

`template-source:` resolves relative to the directory configured as `template.path` in `flapi.yaml`. Sub-paths are honoured:

```
# template.path: sqls/template-source: customers.sql# -> sqls/customers.sqltemplate-source: customers/list.sql# -> sqls/customers/list.sql
```

Absolute paths and remote (`s3://`, `https://`, ...) paths are used as-is.

### Cache Template Override

```
cache:  template-file: customers_cache.sql  # Same resolution rules as template-source.
```

## YAML Best Practices

### 1\. Consistent indentation

```
# 2 spaces throughoutrequest:  - field-name: status    field-in: query    validators:      - type: enum        allowedValues: [active, inactive]
```

### 2\. Comment intent, not mechanics

```
# Customer search endpoint# Cached every 5 minutes, basic auth, read-only.url-path: /customers/search/
```

### 3\. Use descriptive field names

```
- field-name: customer_segment  description: Market segment classification- field-name: minimum_account_balance  description: Minimum balance in USD
```

### 4\. Group related configuration with separators

```
# -- Endpoint -----------------------------------------------url-path: /customers/method: GET# -- Request ------------------------------------------------request:  - field-name: id    field-in: query# -- Caching ------------------------------------------------cache:  enabled: true
```

### 5\. Pull shared config out into common files

```
{{include:auth from ../common/_shared/auth-basic.yaml}}{{include:rate-limit from ../common/_shared/rate-limit-standard.yaml}}
```

### 6\. Always validate enums and ranges

```
validators:  - type: enum    allowedValues: [pending, completed, cancelled]
```

### 7\. Multi-line descriptions for MCP tools

```
mcp-tool:  name: search_orders  description: |    Search orders by status, customer, and date range.    Returns up to 100 orders per page (use limit/offset for more).
```

### 8\. Keep secrets out of YAML

```
auth:  jwt-secret: '${JWT_SECRET}'   # not a literal
```

## Validation & Testing

### Validate YAML syntax

```
flapii endpoints validate /customers/
```

### Inspect resolved paths and includes

```
flapii config paths sqls/customers/list.yaml
```

### Export resolved configuration

```
flapii config export sqls/customers/list.yaml
```

## Troubleshooting

### YAML syntax errors

Check indentation and quoting; YAML is whitespace-sensitive.

### Include not found

```
Error: Include file not found: common/missing.yaml
```

Verify the file exists at the path relative to the including YAML, and that the section name on the left of `from` matches a top-level key in that file.

### Environment variable not resolved

```
Error: Environment variable 'DB_HOST' not found
```

Export the variable and add it to `template.environment-whitelist` in `flapi.yaml`.

### Template path not found

```
Error: Template file not found: customers.sql
```

Confirm `template-source` resolves against `template.path` from `flapi.yaml`.

## Next Steps

*   [Endpoints Overview](/docs/endpoints/overview.md) — complete configuration tutorial
*   [SQL Templating](/docs/concepts/sql-templating.md) — Mustache syntax
*   [Parameters](/docs/endpoints/parameters.md) — query/path/header/body
*   [Validation](/docs/endpoints/validation.md) — the seven validator types
*   [Authentication](/docs/endpoints/authentication.md) — basic, JWT, bearer, OIDC
*   [Write Operations](/docs/endpoints/write-operations.md) — POST/PUT/PATCH/DELETE

([🍪 Cookie Settings](#cookie-settings))

# Agent-Managed flAPI: Dynamic Endpoint Creation

The most powerful thing you can do with flAPI's MCP support is to **let an AI agent reconfigure flAPI itself**. Instead of editing YAML by hand, a developer asks Claude (or any MCP-aware agent) "expose `orders.unfulfilled` as a REST endpoint with caching" — and the agent uses flAPI's built-in `flapi_*` admin tools to inspect the schema, write the SQL, register the endpoint, validate it, hot-reload it, and prime the cache. All over a single MCP session, no `flapi` restart required.

This recipe walks through the full sequence, payload by payload.

## What You'll Build

A flAPI server started with `--config-service`, plus a worked example of an agent driving it through a 7-step create-an-endpoint workflow:

1.  Discover available tables (`flapi_get_schema`)
2.  Confirm the target path is free (`flapi_list_endpoints`)
3.  Register the endpoint config (`flapi_create_endpoint`)
4.  Write the SQL template (`flapi_update_template`)
5.  Validate the template syntax (`flapi_test_template`)
6.  Hot-reload into the running server (`flapi_reload_endpoint`)
7.  Prime the cache (`flapi_refresh_cache`)

The end state: `curl http://localhost:8080/orders/unfulfilled` returns data, and the developer never edited a YAML file directly.

## Prerequisites

*   flAPI installed ([Quickstart](/docs/getting-started/quickstart.md))
*   An MCP-compatible agent (Claude Desktop, Claude Code, or a custom client)
*   A connection already declared in `flapi.yaml` — the agent can create endpoints, but not new connections

## Activating the Config Service

The 19 `flapi_*` admin tools live behind a feature flag. flAPI's standard binary does **not** load them; you must start the server with `--config-service`.

```
# Minimal (no auth — local dev only!)$ ./flapi -c flapi.yaml --config-service# Recommended: protect mutating tools with a bearer token$ export FLAPI_CONFIG_SERVICE_TOKEN="$(openssl rand -hex 32)"$ ./flapi -c flapi.yaml --config-service --config-service-token "$FLAPI_CONFIG_SERVICE_TOKEN"
```

Without the flag, every `flapi_*` call returns:

```
{  "jsonrpc": "2.0",  "id": 1,  "error": {    "code": -32603,    "message": "Tool execution failed: Config tools not available"  }}
```

See [MCP Config Tools](/docs/ai-integration/mcp-config-tools.md) for the full tool catalogue and authentication rules.

**Token hygiene:**

**Never paste the config-service token directly into a chat with the agent.** Keep it in an environment variable (`FLAPI_CONFIG_SERVICE_TOKEN`) and have your MCP bridge inject it into the `Authorization: Bearer …` header on every request. Tokens that land in chat transcripts end up in shared logs, training data exports, and screenshot bug reports. Treat them like database passwords.

## Project Layout

```
orders-api/├── flapi.yaml          # already configured; declares the connection├── data/│   └── orders.parquet└── sqls/    └── (the agent will populate these files)
```

**`flapi.yaml` (pre-existing):**

```
project-name: orders-apiproject-description: Orders REST API, agent-managedtemplate:  path: './sqls'connections:  orders-parquet:    properties:      path: './data/orders.parquet'duckdb:  access_mode: READ_WRITEmcp:  enabled: true
```

Note that `data/orders.parquet` is loaded as a single DuckDB-readable file; the agent will treat it as table `orders`.

## The Agent's Workflow

Every call below is a `POST /mcp/jsonrpc` request. Sessions, headers, and error codes are documented in [MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md). Only the JSON body is shown.

Mutating tools (`flapi_create_endpoint`, `flapi_update_template`, `flapi_reload_endpoint`, `flapi_refresh_cache`) additionally require:

```
Authorization: Bearer $FLAPI_CONFIG_SERVICE_TOKEN
```

### Step 1 — Discover the schema

The agent has no idea what tables are available. The first call is always `flapi_get_schema`.

```
{  "jsonrpc": "2.0",  "id": 1,  "method": "tools/call",  "params": {    "name": "flapi_get_schema",    "arguments": {}  }}
```

**Response (abridged):**

```
{  "jsonrpc": "2.0",  "id": 1,  "result": {    "content": [      {        "type": "text",        "text": "{\"tables\":[{\"name\":\"orders\",\"schema\":\"main\",\"columns\":[{\"name\":\"order_id\",\"type\":\"INTEGER\",\"nullable\":false},{\"name\":\"customer_id\",\"type\":\"INTEGER\",\"nullable\":false},{\"name\":\"status\",\"type\":\"VARCHAR\",\"nullable\":false},{\"name\":\"order_date\",\"type\":\"DATE\",\"nullable\":false},{\"name\":\"total_amount\",\"type\":\"DOUBLE\",\"nullable\":true}]}]}"      }    ]  }}
```

The agent parses the inner JSON and now knows the table is called `orders`, has a `status` column, and that filtering by `status = 'UNFULFILLED'` is a viable strategy.

### Step 2 — Confirm the path is free

Before creating an endpoint, the agent checks that `orders/unfulfilled` is not already taken.

```
{  "jsonrpc": "2.0",  "id": 2,  "method": "tools/call",  "params": {    "name": "flapi_list_endpoints",    "arguments": {}  }}
```

**Response:**

```
{  "jsonrpc": "2.0",  "id": 2,  "result": {    "content": [      {        "type": "text",        "text": "{\"count\": 0, \"endpoints\": []}"      }    ]  }}
```

Zero endpoints — clear to create.

### Step 3 — Register the endpoint

`flapi_create_endpoint` writes a new YAML file under `sqls/`. The agent supplies the URL path, HTTP method, and the filename of the SQL template (which doesn't exist yet — the next step fills it in).

```
{  "jsonrpc": "2.0",  "id": 3,  "method": "tools/call",  "params": {    "name": "flapi_create_endpoint",    "arguments": {      "path": "orders/unfulfilled",      "method": "GET",      "template_source": "orders-unfulfilled.sql"    }  }}
```

**Response:**

```
{  "jsonrpc": "2.0",  "id": 3,  "result": {    "content": [      {        "type": "text",        "text": "{\"status\":\"success\",\"path\":\"orders/unfulfilled\",\"method\":\"GET\",\"template_source\":\"orders-unfulfilled.sql\",\"message\":\"Endpoint created successfully\"}"      }    ]  }}
```

flAPI has now written `sqls/orders-unfulfilled.yaml` containing the basic scaffold (url-path, method, template-source, connection). The endpoint is registered but not yet live — the SQL template doesn't exist.

**Request fields:**

`flapi_create_endpoint` only accepts `path`, `method`, and `template_source`. To add `request:` parameters, validators, caching, or auth, the agent edits the generated YAML using `flapi_update_endpoint` (for top-level switches) or by writing the YAML directly through the [Config Service REST API](/docs/tools/config-service-api.md). For this recipe we keep the endpoint parameter-free and apply caching inline in the YAML once flAPI bootstraps it.

### Step 4 — Write the SQL

Now the agent writes the actual SQL template. The Mustache placeholders are unused here (no parameters), but the template still needs to be valid Mustache.

```
{  "jsonrpc": "2.0",  "id": 4,  "method": "tools/call",  "params": {    "name": "flapi_update_template",    "arguments": {      "endpoint": "orders/unfulfilled",      "content": "-- Unfulfilled orders, newest first\nSELECT\n  order_id,\n  customer_id,\n  status,\n  order_date,\n  total_amount\nFROM orders\nWHERE status = 'UNFULFILLED'\nORDER BY order_date DESC\nLIMIT 500"    }  }}
```

**Response:**

```
{  "jsonrpc": "2.0",  "id": 4,  "result": {    "content": [      {        "type": "text",        "text": "{\"endpoint\":\"orders/unfulfilled\",\"message\":\"Template updated successfully\",\"content_length\":162}"      }    ]  }}
```

The SQL is written to `sqls/orders-unfulfilled.sql` on disk.

### Step 5 — Validate the template

There is no `flapi_validate_endpoint` in flAPI's tool catalogue — validation is split across two read-only tools:

*   `flapi_test_template` — checks Mustache syntax (does every section have a closing tag, etc.)
*   `flapi_expand_template` — renders the Mustache against sample parameters so the agent can preview the SQL before it ever hits DuckDB.

The agent calls both. First, syntax validation:

```
{  "jsonrpc": "2.0",  "id": 5,  "method": "tools/call",  "params": {    "name": "flapi_test_template",    "arguments": { "endpoint": "orders/unfulfilled" }  }}
```

**Response:**

```
{  "jsonrpc": "2.0",  "id": 5,  "result": {    "content": [      {        "type": "text",        "text": "{\"endpoint\":\"orders/unfulfilled\",\"valid\":true,\"message\":\"Template syntax is valid\"}"      }    ]  }}
```

Then optionally an expansion to see the resolved SQL:

```
{  "jsonrpc": "2.0",  "id": 6,  "method": "tools/call",  "params": {    "name": "flapi_expand_template",    "arguments": {      "endpoint": "orders/unfulfilled",      "params": {}    }  }}
```

**Response:**

```
{  "jsonrpc": "2.0",  "id": 6,  "result": {    "content": [      {        "type": "text",        "text": "{\"endpoint\":\"orders/unfulfilled\",\"expanded_sql\":\"-- Unfulfilled orders, newest first\\nSELECT\\n  order_id,\\n  customer_id,\\n  status,\\n  order_date,\\n  total_amount\\nFROM orders\\nWHERE status = 'UNFULFILLED'\\nORDER BY order_date DESC\\nLIMIT 500\",\"status\":\"Template expanded successfully\"}"      }    ]  }}
```

If `flapi_test_template` had returned `valid: false`, the response would have included a `reason` field — the agent can read it and call `flapi_update_template` again with corrected content.

### Step 6 — Hot-reload

The endpoint config and template are on disk, but the running flAPI process hasn't picked them up yet. The agent issues `flapi_reload_endpoint` to trigger an in-place reload — no restart, no downtime for other endpoints.

```
{  "jsonrpc": "2.0",  "id": 7,  "method": "tools/call",  "params": {    "name": "flapi_reload_endpoint",    "arguments": { "path": "orders/unfulfilled" }  }}
```

**Response:**

```
{  "jsonrpc": "2.0",  "id": 7,  "result": {    "content": [      {        "type": "text",        "text": "{\"status\":\"success\",\"path\":\"orders/unfulfilled\",\"message\":\"Endpoint configuration reloaded from disk\",\"original_method\":\"GET\",\"original_template\":\"orders-unfulfilled.sql\"}"      }    ]  }}
```

At this moment `GET /orders/unfulfilled` becomes a live REST route. The server advertises the `listChanged` capability in the `initialize` response but does **not** push `notifications/tools/list_changed` today, so other MCP clients have to re-poll `tools/list` to pick up the new tool (see [MCP Protocol Reference — Notifications](/docs/ai-integration/mcp-protocol.md)).

### Step 7 — Prime the cache

If the endpoint config enabled DuckLake caching (the agent would have added a `cache:` block to the YAML before reload), the first user request would otherwise pay the full SQL latency. The agent triggers a manual refresh so the cache is warm before anyone calls the endpoint.

```
{  "jsonrpc": "2.0",  "id": 8,  "method": "tools/call",  "params": {    "name": "flapi_refresh_cache",    "arguments": { "path": "orders/unfulfilled" }  }}
```

**Response:**

```
{  "jsonrpc": "2.0",  "id": 8,  "result": {    "content": [      {        "type": "text",        "text": "{\"path\":\"orders/unfulfilled\",\"status\":\"Cache refresh triggered\",\"cache_table\":\"orders_unfulfilled_cache\",\"timestamp\":\"1731585600\",\"message\":\"Cache refresh has been scheduled\"}"      }    ]  }}
```

If caching wasn't enabled, this returns `Cache not enabled for this endpoint` with a hint to add a `cache:` block — at which point the agent can either skip the step or update the YAML.

## End State

The developer's terminal now shows:

```
$ curl http://localhost:8080/orders/unfulfilled | jq '.data[0]'{  "order_id": 4117,  "customer_id": 712,  "status": "UNFULFILLED",  "order_date": "2024-11-12",  "total_amount": 1834.50}
```

A complete REST endpoint, sourced from a Parquet file, validated, hot-reloaded, and cache-primed — created by an agent in eight JSON-RPC calls. No `./flapi restart`, no manual YAML editing, no SSH.

## Audit Trail

For each mutation the agent makes, flAPI logs the call (and the bearer token's identity if one was supplied). The agent can request the audit log for cache operations:

```
{  "jsonrpc": "2.0",  "id": 9,  "method": "tools/call",  "params": {    "name": "flapi_get_cache_audit",    "arguments": { "path": "orders/unfulfilled" }  }}
```

**Response:**

```
{  "jsonrpc": "2.0",  "id": 9,  "result": {    "content": [      {        "type": "text",        "text": "{\"path\":\"orders/unfulfilled\",\"cache_table\":\"orders_unfulfilled_cache\",\"audit_log\":[{\"timestamp\":\"1731585600\",\"event\":\"cache_refreshed\",\"status\":\"success\"}]}"      }    ]  }}
```

For project- and template-level changes, inspect the underlying YAML files in `sqls/` — they are still the source of truth, and they live in your VCS history.

## Safety Rails for Production

Letting an agent mutate the live server is powerful **and** dangerous. The standard safety practices:

1.  **Stage agent mutations in a dev project**, then promote the resulting YAML via your normal git review. The agent edits files; treat those edits like any other PR.
2.  **Use a different config-service token per environment.** Dev gets a permissive token; staging and production gate the token behind your secret manager.
3.  **Never expose `--config-service` to the public internet.** Bind it to localhost or behind a VPN. The mutating tools have no per-action ACL beyond the bearer token.
4.  **Audit `flapi_*` calls in your reverse proxy.** Log every mutating method name and the agent that made it. If something breaks at 3 a.m., you want to know who created `orders/unfulfilled` and when.
5.  **Disable the flag entirely in production for static deployments.** If your endpoints are stable, ship the YAML in a Docker image and start flAPI without `--config-service`. The `flapi_*` tools disappear from `tools/list` and there is no agent-write surface to defend.

## The Full Tool Family

This recipe used 7 of the 19 `flapi_*` tools. The rest cover related operations:

| Category | Tools used here | Other tools available |
| --- | --- | --- |
| Discovery (5) | `flapi_get_schema` | `flapi_get_project_config`, `flapi_get_environment`, `flapi_get_filesystem`, `flapi_refresh_schema` |
| Template (4) | `flapi_update_template`, `flapi_test_template`, `flapi_expand_template` | `flapi_get_template` |
| Endpoint (6) | `flapi_list_endpoints`, `flapi_create_endpoint`, `flapi_reload_endpoint` | `flapi_get_endpoint`, `flapi_update_endpoint`, `flapi_delete_endpoint` |
| Cache (4) | `flapi_refresh_cache`, `flapi_get_cache_audit` | `flapi_get_cache_status`, `flapi_run_cache_gc` |

See [MCP Config Tools](/docs/ai-integration/mcp-config-tools.md) for the complete catalogue with arguments, auth requirements, and example payloads.

## See Also

*   **[MCP Config Tools](/docs/ai-integration/mcp-config-tools.md)** — full reference for the `flapi_*` admin tools
*   **[Config Service REST API](/docs/tools/config-service-api.md)** — the HTTP counterpart of these tools (useful for non-MCP automation)
*   **[Configuration Service](/docs/tools/configuration-service.md)** — operator-side docs for `--config-service`, including audit logs and ACL extensions
*   **[MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md)** — session lifecycle, error codes, notifications
*   **[Claude Integration](/docs/ai-integration/claude-integration.md)** — wiring Claude Desktop or Claude Code to a flAPI server with `--config-service`

([🍪 Cookie Settings](#cookie-settings))

# Database Credentials from AWS Secrets Manager

You're running flAPI in production. Two policies make life harder:

1.  **No plaintext credentials in Git.** The compliance team will not approve a `password:` field in any committed YAML, even with environment-variable interpolation.
2.  **30-day rotation.** Every API user's password is rotated on a fixed cadence. Restarting flAPI on each rotation is acceptable; rebuilding the deployment is not.

flAPI's `auth.from-aws-secretmanager` block solves both. The user table lives in AWS Secrets Manager, flAPI loads it into an in-memory DuckDB table at startup, and Basic Auth lookups hit that table on every request. The wire format on AWS is JSON, so rotation is a single `aws secretsmanager update-secret` call.

## How It Works

```
┌─────────────────────────────────────────────────────────┐│                 AWS Secrets Manager                     ││  flapi_endpoint_users                                   ││    { "auth": [                                          ││        { "username": "alice", "password": "...",        ││          "roles": ["read"] },                           ││        { "username": "bob",   "password": "...",        ││          "roles": ["read","write"] }                    ││      ] }                                                │└────────────────────────────┬────────────────────────────┘                             │                             │ 1. On flAPI startup, GetSecretValue                             │    using the DuckDB SECRET named                             │    "flapi_endpoint_users" (TYPE S3).                             ▼┌─────────────────────────────────────────────────────────┐│                    flAPI process                        ││  DuckDB table  api_users(j JSON)                        ││  populated by AwsHelper::refreshSecretJson()            │└────────────────────────────┬────────────────────────────┘                             │                             │ 2. On each request, AuthMiddleware                             │    runs findUserInSecretsTable():                             │      SELECT password, roles                             │      FROM api_users                             │      WHERE username = '<header>'                             ▼┌─────────────────────────────────────────────────────────┐│           HTTP 200  (valid user/pass)                   ││           HTTP 401  (otherwise)                         │└─────────────────────────────────────────────────────────┘
```

Two things worth memorizing:

*   The DuckDB **table** (`secret-table`) is the runtime lookup target. It's an in-memory `CREATE OR REPLACE TABLE <name>(j JSON)` populated from the secret string.
*   The DuckDB **SECRET** (created by `init:` SQL) is _not_ the user list — it's how flAPI authenticates _to AWS_ to read the user list. By convention its name matches `secret-name` (with non-alphanumeric characters replaced by `_`), which is what `tryGetS3AuthParams` looks up.

## The Configuration

This is the pattern from `examples/sqls/taxi/taxi.yaml.inactive`, fleshed out for production:

```
# sqls/customers/customers.yamlurl-path: /customers/method: GETconnection:  - warehousetemplate-source: customers.sqlauth:  enabled: true  type: basic  from-aws-secretmanager:    secret-name: flapi_endpoint_users    secret-table: api_users    region: us-east-1    init: |      CREATE OR REPLACE SECRET flapi_endpoint_users (        TYPE S3,        PROVIDER CREDENTIAL_CHAIN      );
```

Every key here is parsed in `config_manager.cpp` lines 554-582 (`parseEndpointAuth`):

| Key | What it is |
| --- | --- |
| `secret-name` | The AWS Secrets Manager secret identifier (name or ARN). Passed to `GetSecretValueRequest::SetSecretId`. |
| `secret-table` | The DuckDB table name flAPI materializes the secret into. Default is `auth_<sanitized secret-name>`; setting it explicitly is clearer. |
| `region` | AWS region for the Secrets Manager API call. |
| `secret-id` | (Optional) AWS access key ID. Omit to let `PROVIDER CREDENTIAL_CHAIN` use the standard AWS credential chain (IMDS, env vars, `~/.aws/credentials`, IRSA). |
| `secret-key` | (Optional) AWS secret access key. Same caveat. |
| `init` | SQL that runs once on flAPI startup. Use it to define the DuckDB `SECRET` the auth layer uses to talk to AWS. |

If you omit `init:` entirely, flAPI generates a default DuckDB SECRET for you using `secret-id` / `secret-key` (or `PROVIDER CREDENTIAL_CHAIN` if neither is set). The default is built by `ConfigManager::createDefaultAuthInit`. Explicit `init:` blocks override that.

## The Secret Payload

The JSON stored in AWS Secrets Manager must contain an `auth` array of user objects:

```
{  "auth": [    {      "username": "alice",      "password": "5f4dcc3b5aa765d61d8327deb882cf99",      "roles": ["read"]    },    {      "username": "bob",      "password": "9d4e1e23bd5b727046a9e3b4b7db57bd",      "roles": ["read", "write"]    }  ]}
```

Notes from `auth_middleware.cpp::verifyPassword`:

*   A password is treated as an **MD5 hash** when it's exactly 32 hex characters. Anything else is compared as plaintext.
*   Generate MD5 hashes with `echo -n 'my-password' | md5sum`. MD5 is supported for compatibility but is not strong by modern standards — prefer rotating frequently.
*   The `roles` array becomes `auth.roles` (comma-joined) inside SQL templates.

## IAM Permissions

The IAM principal that flAPI runs under (EC2 role, EKS service account via IRSA, or local profile) needs **one** permission:

```
{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "ReadFlapiAuthSecret",      "Effect": "Allow",      "Action": "secretsmanager:GetSecretValue",      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:flapi_endpoint_users-*"    }  ]}
```

Two things to know:

*   AWS appends a 6-character random suffix to the secret's ARN (`-aBcDeF`). Use the `-*` wildcard in the resource ARN.
*   If the secret is KMS-encrypted with a customer-managed key, also grant `kms:Decrypt` on that key.

## Loading Users Into AWS

Initial load:

```
aws secretsmanager create-secret \  --name flapi_endpoint_users \  --region us-east-1 \  --secret-string file://users.json
```

Rotation (just push a new payload):

```
aws secretsmanager update-secret \  --secret-id flapi_endpoint_users \  --region us-east-1 \  --secret-string file://users-2026-05-15.json# Pick up the new payload (the secret is read at startup):systemctl restart flapi    # or: kubectl rollout restart deploy/flapi
```

flAPI reads `secret-name` exactly once during `AuthMiddleware::initialize` — there is no hot reload. A restart (or pod rollout) is what activates the new user list. If you need zero-downtime rotation, run flAPI behind a load balancer and do a rolling restart.

## Test It

Start flAPI and watch for the secret being loaded:

```
flapi --log-level debug --config flapi.yaml# ... Initializing AWS Secrets Manager for endpoint: /customers/# ... Retrieving secret 'flapi_endpoint_users' -> 'flapi_endpoint_users' from AWS Secrets Manager# ... Successfully retrieved secret 'flapi_endpoint_users': *****[247]
```

### Unauthenticated request — 401

```
curl -i http://localhost:8080/customers/# HTTP/1.1 401 Unauthorized# WWW-Authenticate: Basic realm="flAPI"
```

### Wrong password — 401

```
curl -i -u alice:nope http://localhost:8080/customers/# HTTP/1.1 401 Unauthorized
```

### Valid credentials — 200

```
curl -s -u alice:password http://localhost:8080/customers/ | jq .# {#   "data": [ ... ],#   "next": null,#   "total_count": 12# }
```

You can confirm role-based filtering in your SQL template by checking `auth.roles`:

```
-- sqls/customers/customers.sqlSELECT customer_id, company_name, countryFROM warehouse.public.customersWHERE 1=1  -- Only users with the 'write' role see internal-only rows.  AND (       '{{auth.roles}}' LIKE '%write%'    OR is_internal = false  )
```

## Troubleshooting

Run flAPI with `--log-level debug` for diagnostic output. Common failures:

*   **`No AWS auth params found for secret '<name>'`** — the DuckDB SECRET created by `init:` doesn't match the secret name. flAPI sanitizes `secret-name` (non-alphanumeric → `_`) and looks up a DuckDB secret of that name. Make sure your `CREATE SECRET <name>` matches.
*   **`Error retrieving secret '<name>': User: arn:aws:... is not authorized`** — missing IAM permission, or the secret ARN in the policy doesn't include the `-*` suffix.
*   **`Failed to refresh JSON table '<table>'`** — the secret payload isn't valid JSON, or doesn't contain an `auth` array of objects with `username` / `password` / `roles`.
*   **All requests 401, even with correct credentials** — the secret payload is malformed (no `auth` array key) or the password's MD5 hash doesn't match what you generated. Run `echo -n 'my-password' | md5sum` and compare.

## Operational Notes

*   **Rotation cadence**: monthly is the typical target. The `AWSCURRENT` / `AWSPREVIOUS` staging labels on Secrets Manager give you a one-version safety net.
*   **Multiple endpoints, one secret**: flAPI iterates every endpoint with `from-aws-secretmanager` at startup. Sharing one secret across endpoints is fine; flAPI will call `refreshSecretJson` once per endpoint config.
*   **Local development**: use inline `users:` instead of AWS to avoid the IAM dance. The endpoint's `auth:` block selects the lookup path: inline `users:` is tried first, AWS only if no inline match exists (see `AuthMiddleware::authenticateBasic`).
*   **HTTPS**: Basic Auth sends the credential pair base64-encoded — never plaintext-equivalent. Set `enforce-https.enabled: true` and put flAPI behind a TLS terminator.
*   **Connection-level credentials**: this recipe covers _endpoint user_ rotation. If you also need to rotate the _database_ password your connection uses, manage that DuckDB secret directly in the connection's `init:` block — out of scope here, since flAPI doesn't have a built-in helper for it.

## Why a DuckDB SECRET, Not Just IAM?

The `init:` SQL block is the part of the configuration most people stumble on. It creates a DuckDB SECRET, even when the goal is to call AWS Secrets Manager (not S3). Why?

The AWS extension that flAPI's `AwsHelper` uses authenticates via DuckDB's unified secret store. By creating a SECRET named after `secret-name` (sanitized), you give `AwsHelper::tryGetS3AuthParams` a lookup key — that's where it pulls the `key_id`, `secret`, `session_token`, and `region` from. The `TYPE S3` is a bit of a misnomer: it just means "AWS credentials," not "S3 access."

The simplest form, suitable for any environment with an IAM role attached (EC2, ECS, EKS with IRSA, Lambda, or a developer machine with `AWS_PROFILE` set):

```
init: |  CREATE OR REPLACE SECRET flapi_endpoint_users (    TYPE S3,    PROVIDER CREDENTIAL_CHAIN  );
```

`PROVIDER CREDENTIAL_CHAIN` tells DuckDB to walk the standard AWS chain: environment variables, EC2/ECS metadata, IRSA, then `~/.aws/credentials`. This is the production-recommended path — no static keys in config.

For local development without an AWS profile, you can pin the keys explicitly:

```
init: |  CREATE OR REPLACE SECRET flapi_endpoint_users (    TYPE S3,    KEY_ID '{{env.AWS_ACCESS_KEY_ID}}',    SECRET '{{env.AWS_SECRET_ACCESS_KEY}}',    REGION 'us-east-1'  );
```

Or skip the `init:` entirely and let flAPI's `createDefaultAuthInit` generate it from `secret-id` / `secret-key`. The explicit form is preferred — it's easier to grep for and reason about.

## See Also

*   **[Authentication reference](/docs/endpoints/authentication.md)** — the full key list for every auth scheme, including the inline-`users:` shorthand for non-AWS environments.
*   **[Cloud storage and DuckDB secrets](/docs/guides/cloud-storage.md)** — same `CREATE OR REPLACE SECRET ... TYPE S3` pattern, applied to data access instead of auth.
*   **[Multi-Tenant SaaS API with OIDC](/docs/examples/oidc-saas-api.md)** — sibling recipe for delegating authentication entirely to an external IdP.

([🍪 Cookie Settings](#cookie-settings))

# BigQuery + Caching Example

This example shows how to expose BigQuery data as a fast, cost-effective API using flAPI's caching layer.

## Scenario

**Problem:**

*   Marketing team needs a dashboard API
*   Data in BigQuery (100GB table)
*   10,000 API calls per day
*   Direct queries cost $500/day = $15,000/month

**Solution:**

*   Cache aggregated data in DuckDB
*   Refresh hourly
*   Cost: $1.20/day = $36/month
*   **Savings: 99.76%**

## Architecture Flow

## Complete Setup

### 1\. Configuration

**`flapi.yaml`:**

```
project-name: marketing-apiproject-description: Cached BigQuery marketing datatemplate:  path: './sqls'connections:  bigquery-marketing:    init: |      INSTALL 'bigquery' FROM community;      LOAD 'bigquery';    properties:      project_id: 'my-project-id'duckdb:  access_mode: READ_WRITE# DuckLake-backed cache for incremental refresh and snapshotsducklake:  enabled: true  alias: cache  metadata-path: ./data/cache.ducklake  data-path: ./data/cache
```

### 2\. Endpoint Configuration

**`sqls/campaigns.yaml`:**

```
url-path: /campaigns/# Enable cachingcache:  enabled: true  table: campaigns_cache  schema: analytics  schedule: 60m              # Refresh every hour  template-file: campaigns_cache.sql# API parametersrequest:  - field-name: country    field-in: query    description: Filter by country code (e.g., US, DE)    required: false    validators:      - type: string        regex: '^[A-Z]{2}$'  - field-name: campaign_type    field-in: query    description: Campaign type (email, social, search)    required: false    validators:      - type: enum        allowedValues: [email, social, search, display]template-source: campaigns.sqlconnection:  - bigquery-marketing
```

### 3\. Cache Template

**`sqls/campaigns_cache.sql`:**

```
-- This runs every hour to refresh the cache.-- flAPI executes the SELECT and writes the result into the configured-- cache table ({{cache.schema}}.{{cache.table}}) inside DuckLake.SELECT    campaign_id,    campaign_type,    country,    campaign_name,    SUM(clicks) as total_clicks,    SUM(conversions) as total_conversions,    SUM(revenue) as total_revenue,    ROUND(SUM(revenue) / NULLIF(SUM(clicks), 0), 2) as revenue_per_click,    MAX(last_updated) as last_updatedFROM bigquery_scan('{{conn.project_id}}.marketing.campaigns')WHERE date >= CURRENT_DATE - INTERVAL '30 days'  AND active = trueGROUP BY 1, 2, 3, 4
```

### 4\. API Template

**`sqls/campaigns.sql`:**

```
-- This serves API requests from the cache (fast!)-- No BigQuery cost per request. The cache table is fully qualified through-- the DuckLake catalog/schema/table that flAPI injects into the template.SELECT    campaign_id,    campaign_type,    country,    campaign_name,    total_clicks,    total_conversions,    total_revenue,    revenue_per_click,    last_updatedFROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE 1=1{{#params.country}}  AND country = '{{{params.country}}}'{{/params.country}}{{#params.campaign_type}}  AND campaign_type = '{{{params.campaign_type}}}'{{/params.campaign_type}}ORDER BY total_revenue DESCLIMIT 100
```

## Running the Example

### Start flAPI

```
# Set credentialsexport GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json# Start server$ ./flapi -c flapi.yaml✓ Loaded 1 endpoints✓ Cache scheduled: campaigns (60m)✓ Server listening on :8080
```

### Test the API

```
# All campaigns$ curl http://localhost:8080/campaigns/# US campaigns only$ curl "http://localhost:8080/campaigns/?country=US"# Social campaigns in Germany$ curl "http://localhost:8080/campaigns/?country=DE&campaign_type=social"
```

**Response:**

```
{  "data": [    {      "campaign_id": "c_12345",      "campaign_type": "social",      "country": "US",      "campaign_name": "Summer Sale 2024",      "total_clicks": 150000,      "total_conversions": 1200,      "total_revenue": 45000.00,      "revenue_per_click": 0.30,      "last_updated": "2024-01-15 10:30:00"    }  ]}
```

## Cost Analysis

### Without flAPI

```
10,000 API calls/day × $0.05 = $500/dayMonthly cost: $15,000Response time: 2-5 seconds
```

### With flAPI

```
Cache refresh: 24 times/day × $0.05 = $1.20/dayAPI serving: FREE (from cache)Monthly cost: $36Response time: 1-50msSavings: $14,964/month (99.76%)Speed: 1000-10,000x faster
```

## Monitoring Cache

### Check Cache Status

```
$ flapii cache statuscampaigns_cache:  Rows: 12,450  Size: 2.3 MB  Last refresh: 5 minutes ago  API calls since refresh: 1,234
```

### Manual Refresh

```
$ flapii cache refresh campaigns_cacheRefreshing cache: campaigns_cacheSource: BigQuery✓ Query executed (2.3s)✓ 12,450 rows loaded✓ Cache updated
```

## Three cache refresh modes

The example above uses a **full refresh** of an aggregated dataset every hour. flAPI actually supports three refresh patterns, and the right choice depends on table size, how the data mutates, and how fresh the API needs to feel. All three reuse the same `bigquery_scan()` source.

### Mode 1: Full refresh

The simplest mode: every scheduled tick rebuilds the cache table from scratch. Use for small tables, heavy aggregations, or data without a reliable updated-at column.

sqls/campaigns-full.yaml

```
url-path: /campaigns/full/template-source: campaigns.sqlconnection: [bigquery-marketing]cache:  enabled: true  table: campaigns_full_cache  schema: analytics  schedule: 60m  template-file: campaigns_full_cache.sql
```

sqls/campaigns\_full\_cache.sql

```
-- No cursor, no primary key — flAPI rewrites the table on every run.SELECT  campaign_id,  campaign_type,  country,  campaign_name,  clicks,  conversions,  revenue,  last_updatedFROM bigquery_scan('{{conn.project_id}}.marketing.campaigns')WHERE active = true
```

### Mode 2: Incremental append

Add a `cursor:` block (but no `primary-key`) and flAPI exposes `{{cache.previousSnapshotTimestamp}}` to the template. Only rows newer than the last snapshot are scanned, and they're appended to the existing table. Perfect for immutable event streams.

sqls/campaigns-append.yaml

```
url-path: /campaigns/events/template-source: campaigns.sqlconnection: [bigquery-marketing]cache:  enabled: true  table: campaigns_events_cache  schema: analytics  schedule: 5m  template-file: campaigns_events_cache.sql  cursor:    column: last_updated    type: timestamp
```

sqls/campaigns\_events\_cache.sql

```
-- Only fetch rows newer than the previous snapshot.SELECT  campaign_id,  campaign_type,  country,  campaign_name,  clicks,  conversions,  revenue,  last_updatedFROM bigquery_scan('{{conn.project_id}}.marketing.campaigns')WHERE last_updated > COALESCE(  TRY_CAST('{{{cache.previousSnapshotTimestamp}}}' AS TIMESTAMP),  TIMESTAMP '1970-01-01 00:00:00')
```

### Mode 3: Incremental merge

Add both `cursor:` and `primary-key:` and flAPI **upserts** changed rows into the cache. This is what you want for mutable fact tables — campaigns whose budgets get adjusted, products whose stock changes, customers whose emails update. The example below also opts into the full retention controls: keep the 10 most recent snapshots for up to 30 days and allow rollbacks within a 2-day window.

sqls/campaigns-merge.yaml

```
url-path: /campaigns/mutable/template-source: campaigns.sqlconnection: [bigquery-marketing]cache:  enabled: true  table: campaigns_merge_cache  schema: analytics  schedule: 5m  template-file: campaigns_merge_cache.sql  primary-key: [campaign_id]  cursor:    column: last_updated    type: timestamp  retention:    keep-last-snapshots: 10    max-snapshot-age: 30d  rollback-window: 2d
```

sqls/campaigns\_merge\_cache.sql

```
-- Pull updated rows; flAPI merges them onto campaign_id.SELECT  campaign_id,  campaign_type,  country,  campaign_name,  clicks,  conversions,  revenue,  last_updatedFROM bigquery_scan('{{conn.project_id}}.marketing.campaigns')WHERE last_updated > COALESCE(  TRY_CAST('{{{cache.previousSnapshotTimestamp}}}' AS TIMESTAMP),  TIMESTAMP '1970-01-01 00:00:00')
```

### Picking a mode

| Mode | Set | Rebuilds | Good for | Avoid when |
| --- | --- | --- | --- | --- |
| **Full refresh** | nothing | entire table each tick | aggregations, small tables, no updated-at column | tables larger than your refresh budget |
| **Incremental append** | `cursor` | only new rows | event streams, immutable log tables | rows can change after insert |
| **Incremental merge** | `cursor` + `primary-key` | changed rows, upserted | mutable fact tables, slowly-changing dimensions | no reliable row identity |

## Next Steps

*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: Understand cost optimization
*   **[Architecture](/docs/concepts/architecture.md)**: Learn how flAPI achieves performance
*   **[BigQuery Connection Guide](/docs/guides/connections/bigquery.md)**: Detailed BigQuery setup
*   **[Caching Setup Guide](/docs/guides/caching/setup.md)**: Configure caching strategies
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Master template syntax
*   **[Parquet Example](/docs/examples/parquet-api.md)**: Start with local files
*   **[SAP ERP Example](/docs/examples/sap-erp-api.md)**: Enterprise caching patterns
*   **[Deployment](/docs/getting-started/deployment.md)**: Deploy to production

([🍪 Cookie Settings](#cookie-settings))

# Build Your First Complete CRUD API

In this tutorial, you'll build a complete product management API with **Create, Read, Update, and Delete** operations using flAPI. By the end, you'll have a fully functional API that validates input, returns created data, and wraps writes in transactions.

## What You'll Build

A REST API for managing products with these endpoints:

*   `POST /products/` - Create a new product
*   `GET /products/` - List all products
*   `GET /products/:id` - Get a single product
*   `PUT /products/:id` - Update a product
*   `DELETE /products/:id` - Delete a product

## Prerequisites

*   flAPI binary or Docker installed
*   Northwind SQLite database (included in flAPI examples)
*   curl or similar HTTP client

## Step 1: Set Up Your Project

Create the directory structure:

```
mkdir -p my-crud-api/sqlscd my-crud-api
```

flAPI loads endpoint YAML and SQL templates from the same directory configured by `template.path` (here, `./sqls`).

## Step 2: Create a flAPI Configuration File

Create `flapi.yaml`:

```
project-name: my-crud-apiproject-description: Product CRUD API on the Northwind SQLite databasetemplate:  path: './sqls'# SQLite connection to Northwind databaseconnections:  northwind-sqlite:    init: |      INSTALL sqlite;      LOAD sqlite;      ATTACH IF NOT EXISTS './data/northwind.sqlite' AS nw (TYPE sqlite);duckdb:  access_mode: READ_WRITE
```

flAPI automatically picks up every `*.yaml` endpoint file it finds under `template.path` (here, `./sqls`). The CRUD endpoint files in the next steps will all live under that directory.

## Step 3: Create the READ Endpoints

### 3.1: List All Products

Create `sqls/products-list.yaml`:

```
url-path: /products/method: GETdescription: Get all productsrequest: []  # No parameterstemplate-source: products-list.sqlconnection: [northwind-sqlite]
```

Create `sqls/products-list.sql`:

```
SELECT  ProductID as id,  ProductName as name,  SupplierID as supplier_id,  UnitPrice as unit_price,  UnitsInStock as units_in_stock,  Discontinued as discontinuedFROM ProductsORDER BY ProductIDLIMIT 100
```

**Test it:**

```
curl http://localhost:8080/products/
```

### 3.2: Get Single Product

Create `sqls/products-get.yaml`:

```
url-path: /products/:product_idmethod: GETdescription: Get a specific product by IDrequest:  - field-name: product_id    field-in: path    required: true    description: Product ID to retrieve    validators:      - type: int        min: 1template-source: products-get.sqlconnection: [northwind-sqlite]
```

Create `sqls/products-get.sql`:

```
SELECT  ProductID as id,  ProductName as name,  SupplierID as supplier_id,  UnitPrice as unit_price,  UnitsInStock as units_in_stock,  Discontinued as discontinuedFROM ProductsWHERE ProductID = {{{ params.product_id }}}
```

**Test it:**

```
curl http://localhost:8080/products/1
```

## Step 4: Create the CREATE Endpoint

Create `sqls/products-create.yaml`:

```
url-path: /products/method: POSTdescription: Create a new product# Protect writes with JWT (the read endpoints stay public).auth:  enabled: true  type: bearer  jwt-secret: '{{env.PRODUCTS_API_JWT_SECRET}}'  jwt-issuer: my-auth-serveroperation:  type: write  validate-before-write: true  returns-data: true  transaction: true# Drop the cached GET /products/ payload after a successful write so the# next read goes back to SQLite and re-materializes a fresh snapshot.cache:  invalidate-on-write: truerequest:  - field-name: product_name    field-in: body    required: true    description: Product name (1-100 characters)    validators:      - type: string        min: 1        max: 100        preventSqlInjection: true  - field-name: supplier_id    field-in: body    required: true    description: Supplier ID (must exist in database)    validators:      - type: int        min: 1  - field-name: unit_price    field-in: body    required: false    description: Unit price in decimal format    validators:      - type: string        regex: '^\d+(\.\d{1,2})?$'  - field-name: units_in_stock    field-in: body    required: false    description: Number of units in stock    validators:      - type: int        min: 0  - field-name: discontinued    field-in: body    required: false    description: Whether product is discontinued (0 or 1)    validators:      - type: int        min: 0        max: 1    default: "0"template-source: products-create.sqlconnection: [northwind-sqlite]
```

Create `sqls/products-create.sql`:

```
INSERT INTO Products (    ProductName,    SupplierID,    UnitPrice,    UnitsInStock,    Discontinued)VALUES (    '{{{ params.product_name }}}',    {{{ params.supplier_id }}},    {{{ params.unit_price }}},    {{{ params.units_in_stock }}},    COALESCE({{{ params.discontinued }}}, 0))RETURNING  ProductID as id,  ProductName as name,  SupplierID as supplier_id,  UnitPrice as unit_price,  UnitsInStock as units_in_stock,  Discontinued as discontinued
```

**Test it:**

```
curl -X POST http://localhost:8080/products/ \  -H "Content-Type: application/json" \  -d '{    "product_name": "New Widget",    "supplier_id": 1,    "unit_price": "29.99",    "units_in_stock": 100  }'
```

**Response:**

```
{  "returned_data": [    {      "id": 78,      "name": "New Widget",      "supplier_id": 1,      "unit_price": 29.99,      "units_in_stock": 100,      "discontinued": 0    }  ],  "rows_affected": 1}
```

## Step 5: Create the UPDATE Endpoint

Create `sqls/products-update.yaml`:

```
url-path: /products/:product_idmethod: PUTdescription: Update an existing productauth:  enabled: true  type: bearer  jwt-secret: '{{env.PRODUCTS_API_JWT_SECRET}}'  jwt-issuer: my-auth-serveroperation:  type: write  validate-before-write: true  returns-data: true  transaction: truecache:  invalidate-on-write: truerequest:  - field-name: product_id    field-in: path    required: true    validators:      - type: int        min: 1  - field-name: product_name    field-in: body    required: false    validators:      - type: string        min: 1        max: 100        preventSqlInjection: true  - field-name: unit_price    field-in: body    required: false    validators:      - type: string        regex: '^\d+(\.\d{1,2})?$'  - field-name: units_in_stock    field-in: body    required: false    validators:      - type: int        min: 0  - field-name: discontinued    field-in: body    required: false    validators:      - type: int        min: 0        max: 1template-source: products-update.sqlconnection: [northwind-sqlite]
```

Create `sqls/products-update.sql`:

```
UPDATE ProductsSET  ProductName = COALESCE('{{{ params.product_name }}}', ProductName),  UnitPrice = COALESCE({{{ params.unit_price }}}, UnitPrice),  UnitsInStock = COALESCE({{{ params.units_in_stock }}}, UnitsInStock),  Discontinued = COALESCE({{{ params.discontinued }}}, Discontinued)WHERE ProductID = {{{ params.product_id }}}RETURNING  ProductID as id,  ProductName as name,  SupplierID as supplier_id,  UnitPrice as unit_price,  UnitsInStock as units_in_stock,  Discontinued as discontinued
```

**Test it:**

```
curl -X PUT http://localhost:8080/products/78 \  -H "Content-Type: application/json" \  -d '{    "product_name": "Updated Widget",    "unit_price": "34.99"  }'
```

## Step 6: Create the DELETE Endpoint

Create `sqls/products-delete.yaml`:

```
url-path: /products/:product_idmethod: DELETEdescription: Delete a productauth:  enabled: true  type: bearer  jwt-secret: '{{env.PRODUCTS_API_JWT_SECRET}}'  jwt-issuer: my-auth-serveroperation:  type: write  validate-before-write: true  returns-data: false  transaction: truecache:  invalidate-on-write: truerequest:  - field-name: product_id    field-in: path    required: true    validators:      - type: int        min: 1        preventSqlInjection: truetemplate-source: products-delete.sqlconnection: [northwind-sqlite]
```

### Why invalidate on write?

The three write endpoints all set `cache.invalidate-on-write: true`. After a successful `POST`, `PUT`, or `DELETE`, flAPI drops any cached payload for `GET /products/` so the next read re-materializes from SQLite instead of serving the stale snapshot it captured before the mutation. If you'd rather proactively re-run the cache template right after the write instead of waiting for the next request, use `cache.refresh-on-write: true` instead.

### A cached GET for high-traffic reads

If the listing endpoint gets pounded, switch it to a DuckLake-backed cache so hot reads hit DuckDB instead of SQLite. Open `sqls/products-list.yaml` and add a `cache:` block:

```
url-path: /products/method: GETdescription: Get all productscache:  enabled: true  table: products_list_cache  schema: analytics  schedule: 5m  primary-key: [id]request: []template-source: products-list.sqlconnection: [northwind-sqlite]
```

Then update `sqls/products-list.sql` to read from the cache table that flAPI populates for you:

```
SELECT  id,  name,  supplier_id,  unit_price,  units_in_stock,  discontinuedFROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}ORDER BY idLIMIT 100
```

Pair this with the `invalidate-on-write` flags on the write endpoints above and you get the best of both worlds: writes feel instantly consistent because the next `GET` rebuilds the cache, while idle traffic stays millisecond-fast.

Create `sqls/products-delete.sql`:

```
DELETE FROM ProductsWHERE ProductID = {{{ params.product_id }}}
```

**Test it:**

```
curl -X DELETE http://localhost:8080/products/78
```

**Response:**

```
{  "rows_affected": 1}
```

## Step 7: Run the Server

Start your flAPI server:

```
flapi -c flapi.yaml
```

Or with Docker:

```
docker run -p 8080:8080 \  -v $(pwd):/config \  ghcr.io/datazoode/flapi -c /config/flapi.yaml
```

## Step 8: Complete API Walkthrough

### Create a Product

```
curl -X POST http://localhost:8080/products/ \  -H "Content-Type: application/json" \  -d '{    "product_name": "Excellent Widget",    "supplier_id": 2,    "unit_price": "49.99",    "units_in_stock": 500  }'
```

### Read All Products

```
curl http://localhost:8080/products/
```

### Read Specific Product

```
curl http://localhost:8080/products/78
```

### Update Product

```
curl -X PUT http://localhost:8080/products/78 \  -H "Content-Type: application/json" \  -d '{    "units_in_stock": 750,    "unit_price": "54.99"  }'
```

### Delete Product

```
curl -X DELETE http://localhost:8080/products/78
```

## Validation in Action

Try invalid inputs to see validation errors:

```
# Invalid: product_name too shortcurl -X POST http://localhost:8080/products/ \  -H "Content-Type: application/json" \  -d '{    "product_name": "",    "supplier_id": 1  }'# Response:{  "error": {    "field": "product_name",    "message": "String must be at least 1 character"  }}
```

```
# Invalid: price formatcurl -X POST http://localhost:8080/products/ \  -H "Content-Type: application/json" \  -d '{    "product_name": "Widget",    "supplier_id": 1,    "unit_price": "not-a-number"  }'# Response:{  "error": {    "field": "unit_price",    "message": "Invalid format: expected decimal number (e.g., 19.99)"  }}
```

## Next Steps

### Add Security

Protect write endpoints with authentication. Each endpoint takes an `auth` block whose `type` is one of `basic`, `bearer` (also used for JWT bearer tokens — set `jwt-secret` to enable JWT validation), or `oidc`:

```
auth:  enabled: true  type: bearer  jwt-secret: '{{env.JWT_SECRET}}'  jwt-issuer: my-auth-server
```

### Add Rate Limiting

Prevent abuse:

```
rate-limit:  max: 100  interval: 60  # Max 100 requests per minute
```

### Row-Level Security

Restrict admins to all rows and other users to their own data. The authenticated username comes from the `auth.username` context variable, and `auth.roles` is a comma-joined string you can match with `LIKE`:

```
UPDATE ProductsSET ProductName = '{{{ params.product_name }}}'WHERE ProductID = {{{ params.product_id }}}  AND (    '{{{ auth.roles }}}' LIKE '%admin%'    OR owner_username = '{{{ auth.username }}}'  )
```

### Expand Validation

Add more complex validators for your domain:

```
- field-name: discount_percentage  validators:    - type: int      min: 0      max: 100
```

## Learn More

*   [Write Operations Reference](/docs/endpoints/write-operations.md)
*   [Validation](/docs/endpoints/write-operations.md#validation)
*   [Security Best Practices](/docs/endpoints/write-operations.md#security-considerations)
*   [Authentication schemes](/docs/endpoints/authentication.md) — JWT, basic, bearer, and OIDC details
*   [Caching strategy](/docs/concepts/caching-strategy.md) — how DuckLake snapshots and invalidation fit together

([🍪 Cookie Settings](#cookie-settings))

# Pick a DuckLake Cache Refresh Mode

You have a `products` table sitting in Parquet on local disk (or any object store DuckDB can read). You want a fast, snapshot-aware API in front of it. Which refresh mode should you pick?

flAPI doesn't have a `strategy:` knob. The refresh mode is **derived** from your cache block — specifically the combination of `cursor` and `primary-key` — per `CacheManager::determineCacheMode` in `cache_manager.cpp`:

```
no cursor                 -> fullcursor only               -> appendcursor + primary-key      -> merge
```

This recipe takes the **same** Parquet backend and shows all three options side-by-side, with the trade-offs spelled out so you can pick the cheapest mode that still answers your business question.

## The Common Backend

Every variant below uses the same connection. We pin to Parquet because it makes the example deterministic — every variant has identical I/O up to the `WHERE` clause.

```
# flapi.yamlproject-name: products-apiproject-description: Three refresh modes against the same Parquet products tabletemplate:  path: './sqls'connections:  products-parquet:    properties:      path: './data/products.parquet'duckdb:  access_mode: READ_WRITE  threads: 8ducklake:  enabled: true  alias: cache  metadata-path: ./data/cache.ducklake  data-path: ./data/cache  retention:    keep-last-snapshots: 10    max-snapshot-age: 30d  compaction:    enabled: true    schedule: '@daily'  scheduler:    enabled: true    scan-interval: 5m
```

The `ducklake.alias: cache` value becomes `{{cache.catalog}}` everywhere the refresh template runs.

Assume the Parquet file has these columns:

| Column | Type | Notes |
| --- | --- | --- |
| `id` | bigint | Stable primary key |
| `sku` | varchar | Vendor SKU |
| `name` | varchar | Display name |
| `category` | varchar | e.g. `electronics`, `apparel` |
| `price_cents` | bigint | Current price |
| `created_at` | timestamp | Row insert time, never updated |
| `updated_at` | timestamp | Bumped on any change |
| `deleted_at` | timestamp | NULL for live rows, set on soft delete |

## Mode 1: Full Refresh

**When**: small reference tables (under a few million rows), schema is unstable, you don't trust your `updated_at` column, or you simply want the dumbest possible refresh.

**Triggered by**: no `cursor`. flAPI picks `full` automatically.

### Cache configuration

```
# sqls/products/products-rest.yamlurl-path: /products/method: GETconnection:  - products-parquettemplate-source: products.sqlcache:  enabled: true  table: products_cache  schema: catalog  schedule: 1h  retention:    keep-last-snapshots: 5    max-snapshot-age: 7d
```

No `cursor`, no `primary-key`. `CacheManager::determineCacheMode` returns `"full"`.

### Refresh SQL

```
-- sqls/products/products_cache.sql-- Mode: full. Rebuild the cache table from scratch on every schedule.CREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} ASSELECT    id,    sku,    name,    category,    price_cents,    created_at,    updated_at,    deleted_at,    CURRENT_TIMESTAMP        AS cache_updated_at,    '{{cache.snapshotId}}'   AS cache_snapshot_idFROM read_parquet('{{conn.path}}')WHERE deleted_at IS NULLORDER BY id;
```

Notes:

*   We do **not** reference `{{cache.previousSnapshotTimestamp}}` — full mode has no concept of "since last time".
*   The cache template runs unconditionally for the full mode; no mode flag is exposed to the template.

### Cost characteristic

`O(N)` per refresh. Every snapshot scans the entire Parquet file. Cheap with a few million rows, expensive at a billion.

### Caveats

*   Every refresh produces a fully-rewritten DuckLake snapshot, so retention rules will eat through disk faster than incremental modes.
*   API readers see snapshot isolation, but you cannot answer "what changed since yesterday" from the cache itself — only from the audit table.

## Mode 2: Incremental Append

**When**: append-only data (events, logs, orders that never get updated in place). You have a monotonically increasing timestamp or ID and don't care about updates or deletes.

**Triggered by**: `cursor` present, `primary-key` absent. flAPI picks `append`.

### Cache configuration

```
# sqls/products/products-rest.yamlurl-path: /products/method: GETconnection:  - products-parquettemplate-source: products.sqlcache:  enabled: true  table: products_cache  schema: catalog  schedule: 5m  cursor:    column: created_at    type: timestamp  retention:    keep-last-snapshots: 20    max-snapshot-age: 14d
```

### Refresh SQL

```
-- sqls/products/products_cache.sql-- Mode: append. Pull rows created since the previous snapshot.INSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT    id,    sku,    name,    category,    price_cents,    created_at,    updated_at,    deleted_at,    CURRENT_TIMESTAMP        AS cache_updated_at,    '{{cache.snapshotId}}'   AS cache_snapshot_idFROM read_parquet('{{conn.path}}')WHERE {{cache.cursorColumn}} > TIMESTAMP '{{cache.previousSnapshotTimestamp}}'  AND deleted_at IS NULL;
```

On the very first run, `{{cache.previousSnapshotTimestamp}}` is empty, so most teams ship a small bootstrap branch:

```
{{#if cache.previousSnapshotTimestamp}}-- IncrementalINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT ... FROM read_parquet('{{conn.path}}')WHERE created_at > TIMESTAMP '{{cache.previousSnapshotTimestamp}}';{{else}}-- First loadCREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} ASSELECT ... FROM read_parquet('{{conn.path}}');{{/if}}
```

### Cost characteristic

`O(delta)` per refresh — only new rows are scanned and inserted. Cache grows monotonically until retention expires old snapshots.

### Caveats

*   **No update detection.** If a row's `price_cents` changes in Parquet, the cache will not see it. Append mode only watches the cursor.
*   **No delete detection.** Hard deletes upstream are invisible. Soft deletes (`deleted_at` set) can be filtered at read time on the API side.
*   Pick `cursor.column` carefully: it must be **monotonically non-decreasing**. `created_at` is safe; `updated_at` is not (it can move backwards relative to insertion order during backfills).

## Mode 3: Incremental Merge / Upsert

**When**: mutable rows. You have a reliable primary key, a trustworthy `updated_at` column, and you need the cache to reflect upstream changes.

**Triggered by**: both `cursor` **and** `primary-key`. flAPI picks `merge`.

### Cache configuration

```
# sqls/products/products-rest.yamlurl-path: /products/method: GETconnection:  - products-parquettemplate-source: products.sqlcache:  enabled: true  table: products_cache  schema: catalog  schedule: 1m  primary-key: [id]  cursor:    column: updated_at    type: timestamp  rollback-window: 2d  retention:    keep-last-snapshots: 30    max-snapshot-age: 7d  delete-handling: soft
```

### Refresh SQL

```
-- sqls/products/products_cache.sql-- Mode: merge. Upsert by primary key, using updated_at as the cursor.MERGE INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}} AS tgtUSING (    SELECT        id,        sku,        name,        category,        price_cents,        created_at,        updated_at,        deleted_at,        CURRENT_TIMESTAMP        AS cache_updated_at,        '{{cache.snapshotId}}'   AS cache_snapshot_id    FROM read_parquet('{{conn.path}}')    WHERE {{cache.cursorColumn}} > TIMESTAMP '{{cache.previousSnapshotTimestamp}}') AS srcON tgt.id = src.idWHEN MATCHED THEN UPDATE SET    sku               = src.sku,    name              = src.name,    category          = src.category,    price_cents       = src.price_cents,    updated_at        = src.updated_at,    deleted_at        = src.deleted_at,    cache_updated_at  = src.cache_updated_at,    cache_snapshot_id = src.cache_snapshot_idWHEN NOT MATCHED THEN INSERT VALUES (    src.id, src.sku, src.name, src.category, src.price_cents,    src.created_at, src.updated_at, src.deleted_at,    src.cache_updated_at, src.cache_snapshot_id);
```

Note we use `updated_at` as the cursor (not `created_at`) — that's the whole point of merge mode. `{{cache.primaryKeys}}` is available if you want to drive a generic template across multiple endpoints.

### Cost characteristic

`O(updates + inserts)` per refresh. Cheap as long as your upstream system updates only a small fraction of rows per cycle.

### Caveats

*   **Hard deletes still need help.** A `MERGE` only sees rows that come back from the source query — physically removed rows are invisible. If you can't trust upstream to soft-delete, you must periodically reconcile (e.g. a daily full sweep). flAPI's `delete-handling: soft|hard` controls only how the cache stores deletes it has been told about; it does not invent delete detection.
*   The `cursor.column` must be **set whenever a row changes**. Stale `updated_at` values are silent data corruption in merge mode.

## Side-by-Side Comparison

| Property | Full | Append | Merge |
| --- | --- | --- | --- |
| Detects inserts | yes | yes | yes |
| Detects updates | yes | no  | yes |
| Detects deletes | yes (via re-scan) | no  | only soft deletes |
| Scan cost per refresh | `O(N)` | `O(delta)` | `O(delta)` |
| Cache write cost | rewrite entire table | append rows | upsert matching rows |
| Required source columns | none | a monotonic cursor | primary key + cursor |
| Template complexity | lowest | medium (bootstrap branch) | highest (`MERGE` statement) |
| Triggered by | no `cursor` | `cursor` only | `cursor` + `primary-key` |

**Rule of thumb**: start with **full** for anything under ~1M rows. Move to **append** when refresh time becomes annoying. Move to **merge** when you actually need to see updates.

## Retention, Rollback, and Delete Handling

The same DuckLake controls apply to all three modes:

```
cache:  enabled: true  table: products_cache  schema: catalog  schedule: 1m  primary-key: [id]  cursor:    column: updated_at    type: timestamp  retention:    keep-last-snapshots: 30   # keep the 30 most recent snapshots    max-snapshot-age: 7d      # ... or anything younger than 7 days  rollback-window: 2d         # time-travel window for ad-hoc reads  delete-handling: soft       # soft | hard
```

What each does:

*   **`retention.keep-last-snapshots`** — flAPI invokes `ducklake_expire_snapshots(..., versions => ARRAY[0:N])` after each refresh.
*   **`retention.max-snapshot-age`** — flAPI invokes `ducklake_expire_snapshots(..., older_than => CURRENT_TIMESTAMP - INTERVAL '...')`.
*   **`rollback-window`** — snapshots inside this window remain queryable for time-travel even if they would otherwise be expired.
*   **`delete-handling: soft`** — deleted rows are tombstoned in the cache (kept with a deletion marker).
*   **`delete-handling: hard`** — deleted rows are physically removed.

Every refresh — successful or failed — is recorded in `<catalog>.audit.sync_events` with `sync_type` set to `full`, `append`, `merge`, or `garbage_collection`. The audit table is created automatically by `CacheManager::initializeAuditTables`.

## Reading the Cache from the API

Regardless of mode, the API endpoint template reads from the **cache table**, not the source. That's how you get sub-5ms responses.

```
-- sqls/products/products.sqlSELECT    id,    sku,    name,    category,    price_centsFROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE 1=1  {{#if request.category}}    AND category = '{{request.category}}'  {{/if}}  {{#if request.id}}    AND id = {{request.id}}  {{/if}}  AND deleted_at IS NULLORDER BY idLIMIT {{request.limit}};
```

Three things to notice:

1.  `{{cache.catalog}}.{{cache.schema}}.{{cache.table}}` is the same triple injected during refresh, so the API and refresh templates stay in sync via configuration, not copy-paste.
2.  Filtering `deleted_at IS NULL` at read time is what gives `delete-handling: soft` its name — the rows are still in the cache, just hidden.
3.  None of the `{{cache.previousSnapshot*}}` variables are populated on read-time requests. They only exist during a refresh. Don't reference them in API templates.

## Cache Template Variables Reference

These are the variables flAPI injects into your cache refresh template, verified against `src/sql_template_processor.cpp`:

| Variable | Populated when |
| --- | --- |
| `{{cache.catalog}}` | always (from `ducklake.alias`) |
| `{{cache.schema}}` | always (defaults to `main`) |
| `{{cache.table}}` | always |
| `{{cache.schedule}}` | when `cache.schedule` is set |
| `{{cache.snapshotId}}` | once a current snapshot exists |
| `{{cache.snapshotTimestamp}}` | once a current snapshot exists |
| `{{cache.previousSnapshotId}}` | once a previous snapshot exists (i.e. second refresh onwards) |
| `{{cache.previousSnapshotTimestamp}}` | once a previous snapshot exists |
| `{{cache.cursorColumn}}` | when `cursor` is configured |
| `{{cache.cursorType}}` | when `cursor` is configured |
| `{{cache.primaryKeys}}` | when `primary-key` is configured (comma-separated) |

Any other `{{cache.*}}` you see in a tutorial is wrong. There is no `{{cache.strategy}}`, no `{{cache.lastRefresh}}`, no `{{cache.delta}}`.

## Choosing in 30 Seconds

Answer these in order and stop at the first "yes":

1.  Is the table under ~1M rows or do you not have a trustworthy timestamp? **Use full.**
2.  Is the table append-only (or do you genuinely not care about updates)? **Use append.**
3.  Do you have a stable primary key and a `updated_at` that is bumped on every change? **Use merge.**

If you can't answer "yes" to any of these, you have a data-model problem upstream that no cache can hide. Fix the source.

## Next Steps

*   [Caching Strategy concept page](/docs/concepts/caching-strategy.md) — the theory behind DuckLake and the three modes.
*   [Caching setup guide](/docs/guides/caching/setup.md) — step-by-step initial configuration.
*   [BigQuery + Caching example](/docs/examples/bigquery-caching.md) — the same modes applied to a per-query-cost backend.
*   [Config Service REST API](/docs/tools/config-service-api.md) — trigger a manual cache refresh without restarting the server.

([🍪 Cookie Settings](#cookie-settings))

# Google Sheets Product Catalog API

**Extension Credit:**

This example uses the **[gsheets extension](https://duckdb.org/community_extensions/extensions/gsheets.html)** by [archiewood](https://github.com/archiewood) and [mharrisb1](https://github.com/mharrisb1). Thanks to the DuckDB community for enabling Google Sheets integration!

Build a production-ready product catalog API backed by Google Sheets. Perfect for teams where non-technical staff need to manage product data without touching code or databases.

## Use Case

**Scenario:** A growing e-commerce company needs a product catalog API, but:

*   Marketing team updates product descriptions daily
*   Pricing changes multiple times per week
*   Developers are focused on core features
*   No time to build an admin panel

**Solution:** Use Google Sheets as the "database" that marketing can edit directly, with flAPI providing the REST API layer.

## Architecture

**Benefits:**

*   ✅ Marketing updates products instantly
*   ✅ No admin panel to build/maintain
*   ✅ Version history built-in (Google Sheets)
*   ✅ Collaborative editing
*   ✅ Millisecond API response time (with caching)

## Step 1: Create the Spreadsheet

### Products Sheet

Create a Google Sheet with this structure:

**URL:** `https://docs.google.com/spreadsheets/d/1xKXvY8mHnEqWpLzRtBgJ9cNdFhPqSaUo/edit`

| sku | name | description | price | category | in\_stock | image\_url | created\_date |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SKU-001 | Wireless Mouse | Ergonomic wireless mouse with 3 buttons | 29.99 | Electronics | TRUE | https://... | 2024-01-15 |
| SKU-002 | Laptop Stand | Adjustable aluminum laptop stand | 49.99 | Accessories | TRUE | https://... | 2024-01-16 |
| SKU-003 | USB-C Cable | 6ft USB-C to USB-C cable | 12.99 | Cables | FALSE | https://... | 2024-01-17 |
| SKU-004 | Desk Lamp | LED desk lamp with touch control | 39.99 | Furniture | TRUE | https://... | 2024-01-18 |
| SKU-005 | Keyboard | Mechanical keyboard with RGB | 89.99 | Electronics | TRUE | https://... | 2024-01-19 |

### Categories Sheet

Add a second sheet named "Categories":

| category\_id | category\_name | description |
| --- | --- | --- |
| CAT-001 | Electronics | Electronic devices and gadgets |
| CAT-002 | Accessories | Computer and desk accessories |
| CAT-003 | Cables | Charging and data cables |
| CAT-004 | Furniture | Office furniture and ergonomics |

**Share Settings:**

1.  Click "Share" → Add your Google service account email
2.  Give "Viewer" permission (read-only for production)
3.  Copy the Spreadsheet ID from the URL: `1xKXvY8mHnEqWpLzRtBgJ9cNdFhPqSaUo`

## Step 2: Configure flAPI

### Main Configuration

flapi.yaml

```
project-name: sheets-product-catalogproject-description: Product catalog API backed by Google Sheetstemplate:  path: './sqls'  environment-whitelist:    - '^GOOGLE_.*'connections:  product-sheets:    init: |      -- Install Google Sheets extension      INSTALL gsheets FROM community;      LOAD gsheets;      -- Authenticate with service account      CREATE SECRET (        TYPE gsheet,        PROVIDER access_token,        TOKEN '{{env.GOOGLE_SHEETS_ACCESS_TOKEN}}'      );    properties:      spreadsheet_id: '1xKXvY8mHnEqWpLzRtBgJ9cNdFhPqSaUo'
```

### Environment Variables

.env

```
GOOGLE_SHEETS_ACCESS_TOKEN=ya29.a0AfH6SMBx...your-token-here
```

## Step 3: Create API Endpoints

### Endpoint 1: List All Products

The API SQL reads from the cache table rather than hitting Google Sheets on every request. flAPI injects the cache catalog, schema, and table names into the template via the `cache.*` context, so the query always points at the materialized snapshot:

sqls/products.sql

```
SELECT  sku,  name,  description,  price,  category,  in_stock,  image_url,  created_dateFROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE 1=1{{#params.category}}  AND LOWER(category) = LOWER('{{{params.category}}}'){{/params.category}}{{#params.in_stock}}  AND in_stock = {{{params.in_stock}}}{{/params.in_stock}}{{#params.min_price}}  AND price >= {{{params.min_price}}}{{/params.min_price}}{{#params.max_price}}  AND price <= {{{params.max_price}}}{{/params.max_price}}{{#params.search}}  AND (    LOWER(name) LIKE LOWER('%{{{params.search}}}%')    OR LOWER(description) LIKE LOWER('%{{{params.search}}}%')  ){{/params.search}}ORDER BY{{#params.sort_by}}  CASE '{{{params.sort_by}}}'    WHEN 'price_asc' THEN price    WHEN 'price_desc' THEN -price    WHEN 'name' THEN name    ELSE created_date  END{{/params.sort_by}}{{^params.sort_by}}  created_date DESC{{/params.sort_by}}{{#params.limit}}LIMIT {{{params.limit}}}{{/params.limit}}{{^params.limit}}LIMIT 100{{/params.limit}}
```

Upgrade the endpoint to an **incremental-merge** cache so flAPI only pulls rows from the sheet whose `updated_at` is newer than the last snapshot, and merges them on the `sku` primary key:

sqls/products.yaml

```
url-path: /products/description: Get product catalog from Google Sheetstemplate-source: products.sqlconnection:  - product-sheetscache:  enabled: true  table: products_cache  schema: analytics  schedule: 5m              # Refresh every 5 minutes  template-file: products_cache.sql  # Incremental-merge configuration  primary-key: [sku]        # row identity for UPSERTs  cursor:    column: updated_at      # only pull rows newer than the last snapshot    type: timestamp  # DuckLake snapshot retention  retention:    keep-last-snapshots: 5    max-snapshot-age: 14drequest:  - field-name: category    field-in: query    description: Filter by category name    required: false    validators:      - type: string        max: 50  - field-name: in_stock    field-in: query    description: Filter by stock status ("true" or "false")    required: false    validators:      - type: enum        allowedValues: ["true", "false"]  - field-name: min_price    field-in: query    description: Minimum price filter    required: false    validators:      - type: int        min: 0  - field-name: max_price    field-in: query    description: Maximum price filter    required: false    validators:      - type: int        min: 0  - field-name: search    field-in: query    description: Search in name and description    required: false    validators:      - type: string        max: 100  - field-name: sort_by    field-in: query    description: Sort order    required: false    validators:      - type: enum        allowedValues: [price_asc, price_desc, name, date]  - field-name: limit    field-in: query    description: Maximum results    required: false    validators:      - type: int        min: 1        max: 500
```

### Cache Template

The cache template runs on every scheduled tick. When `primary-key` and `cursor` are configured, flAPI exposes `{{cache.previousSnapshotTimestamp}}` to the template — use it to pull only the rows that changed since the last materialization:

sqls/products\_cache.sql

```
-- Incremental merge: only pull rows updated since the last snapshot.-- On the very first run the previous timestamp is empty, so we fall back-- to a full sweep with a sentinel date.SELECT  sku,  name,  description,  price,  category,  in_stock,  image_url,  created_date,  updated_atFROM read_gsheet('{{{conn.spreadsheet_id}}}', sheet='Products')WHERE updated_at > COALESCE(  TRY_CAST('{{{cache.previousSnapshotTimestamp}}}' AS TIMESTAMP),  TIMESTAMP '1970-01-01 00:00:00')
```

**Pick the right refresh mode:**

flAPI supports three refresh patterns: **full refresh** (no cursor, no primary key — rewrite the whole table each run), **incremental append** (cursor only — fast inserts for immutable event streams), and **incremental merge** (cursor + primary key — what we use here, perfect for mutable rows like a product catalog where prices and stock change in place). See [Caching Strategy](/docs/concepts/caching-strategy.md) for the tradeoffs.

### Endpoint 2: Single Product

sqls/product\_detail.sql

```
SELECT  sku,  name,  description,  price,  category,  in_stock,  image_url,  created_dateFROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE sku = '{{{params.sku}}}'LIMIT 1
```

sqls/product\_detail.yaml

```
url-path: /products/:sku/template-source: product_detail.sqlconnection:  - product-sheetscache:  enabled: true  table: products_cache  schema: analytics  schedule: 5m  template-file: products_cache.sql  primary-key: [sku]  cursor:    column: updated_at    type: timestamprequest:  - field-name: sku    field-in: path    description: Product SKU    required: true    validators:      - type: string        regex: '^SKU-[0-9]{3}$'
```

### Endpoint 3: Categories

sqls/categories.sql

```
SELECT  category_id,  category_name,  descriptionFROM read_gsheet('{{{conn.spreadsheet_id}}}', sheet='Categories')ORDER BY category_name
```

sqls/categories.yaml

```
url-path: /categories/template-source: categories.sqlconnection:  - product-sheetscache:  enabled: true  table: categories_cache  schema: analytics  schedule: 30m              # Refresh every 30 minutes  template-file: categories_cache.sql
```

sqls/categories\_cache.sql

```
SELECT * FROM read_gsheet('{{{conn.spreadsheet_id}}}', sheet='Categories')
```

### Endpoint 4: Products by Category (with counts)

sqls/products\_by\_category.sql

```
WITH category_products AS (  SELECT    category,    COUNT(*) as product_count,    COUNT(CASE WHEN in_stock THEN 1 END) as in_stock_count,    AVG(price) as avg_price,    MIN(price) as min_price,    MAX(price) as max_price  FROM read_gsheet('{{{conn.spreadsheet_id}}}', sheet='Products')  GROUP BY category)SELECT  c.category_id,  c.category_name,  c.description,  COALESCE(cp.product_count, 0) as product_count,  COALESCE(cp.in_stock_count, 0) as in_stock_count,  ROUND(COALESCE(cp.avg_price, 0), 2) as avg_price,  COALESCE(cp.min_price, 0) as min_price,  COALESCE(cp.max_price, 0) as max_priceFROM read_gsheet('{{{conn.spreadsheet_id}}}', sheet='Categories') cLEFT JOIN category_products cp ON c.category_name = cp.categoryORDER BY c.category_name
```

sqls/products\_by\_category.yaml

```
url-path: /categories/summary/template-source: products_by_category.sqlconnection:  - product-sheetscache:  enabled: true  table: products_by_category_cache  schema: analytics  schedule: 10m  template-file: products_cache.sql
```

## Step 4: Deploy & Test

### Start flAPI

```
# Export your Google Sheets tokenexport GOOGLE_SHEETS_ACCESS_TOKEN="ya29.a0AfH6SMBx..."# Run flAPIdocker run -p 8080:8080 \  -v $(pwd)/flapi.yaml:/config/flapi.yaml \  -v $(pwd)/sqls:/config/sqls \  -e GOOGLE_SHEETS_ACCESS_TOKEN \  ghcr.io/datazoode/flapi:latest
```

### Test the APIs

```
# Get all productscurl http://localhost:8080/products/# Filter by categorycurl http://localhost:8080/products/?category=Electronics# Search productscurl 'http://localhost:8080/products/?search=wireless'# Price range + in stockcurl 'http://localhost:8080/products/?min_price=20&max_price=50&in_stock=true'# Single productcurl http://localhost:8080/products/SKU-001/# Categories with statscurl http://localhost:8080/categories/summary/
```

### Sample Response

```
{  "data": [    {      "sku": "SKU-001",      "name": "Wireless Mouse",      "description": "Ergonomic wireless mouse with 3 buttons",      "price": 29.99,      "category": "Electronics",      "in_stock": true,      "image_url": "https://example.com/images/mouse.jpg",      "created_date": "2024-01-15"    }  ],  "row_count": 1}
```

## Step 5: Integration with Website

### JavaScript Example

```
// Fetch products for product listing pageasync function loadProducts(category = null, inStock = true) {  const params = new URLSearchParams();  if (category) params.append('category', category);  if (inStock) params.append('in_stock', 'true');  params.append('limit', '20');    const response = await fetch(    `https://api.yoursite.com/products/?${params}`  );  const data = await response.json();    return data.data;}// Display productsloadProducts('Electronics', true).then(products => {  products.forEach(product => {    console.log(`${product.name} - $${product.price}`);  });});
```

### React Hook Example

```
import { useState, useEffect } from 'react';function useProducts(category, inStock) {  const [products, setProducts] = useState([]);  const [loading, setLoading] = useState(true);  useEffect(() => {    const params = new URLSearchParams();    if (category) params.append('category', category);    params.append('in_stock', inStock.toString());    fetch(`https://api.yoursite.com/products/?${params}`)      .then(res => res.json())      .then(data => {        setProducts(data.data);        setLoading(false);      });  }, [category, inStock]);  return { products, loading };}// Usagefunction ProductList() {  const { products, loading } = useProducts('Electronics', true);    if (loading) return <div>Loading...</div>;    return (    <ul>      {products.map(p => (        <li key={p.sku}>{p.name} - ${p.price}</li>      ))}    </ul>  );}
```

## Performance Analysis

### Without Caching

```
Direct Google Sheets queries:- Response time: ~500-800ms- Rate limit: 100 requests/100 seconds- Cost: $0 (within quota)
```

### With flAPI Caching

```
Cached queries:- Response time: 1-10ms (100x faster!)- Rate limit: Unlimited- Cost: $0- Cache refresh: Every 5 minutes
```

**Performance improvement: 50-500x faster**

## Real-World Metrics

This exact setup is used by a Shopify store with:

*   **500 products** in Google Sheets
*   **10,000 API calls/day** from website
*   **~2ms avg response time**
*   **Marketing team updates 20+ products/day**
*   **Zero database costs**

## Scaling Considerations

### Current Limits

*   Products: Up to ~10,000 (Google Sheets: 5M cells)
*   API throughput: ~1,000 req/sec (with caching)
*   Cache storage: ~100MB for 10k products

### When to Migrate

Consider migrating to PostgreSQL/BigQuery when:

*   ❌ Products > 50,000
*   ❌ Need transaction support
*   ❌ Complex inventory management
*   ❌ Real-time stock updates

**Good news:** With flAPI, migration is seamless - just change the connection config!

## Troubleshooting

### Issue: "Authentication failed"

```
# Verify tokenecho $GOOGLE_SHEETS_ACCESS_TOKEN# Regenerate token# 1. Go to Google Cloud Console# 2. Create new service account key# 3. Update environment variable
```

### Issue: "Sheet not found"

```
# Verify sheet name (case-sensitive!)properties:  spreadsheet_id: '1xKXvY8mHnEqWpLzRtBgJ9cNdFhPqSaUo'  sheet_name: 'Products'  # Must match exactly
```

### Issue: Slow first request

This is normal - first cache refresh takes 500ms. Subsequent requests are instant.

```
# Set a short schedule so the cache warms quickly after server startcache:  enabled: true  table: products_cache  schema: analytics  schedule: 5m  template-file: products_cache.sql
```

## Next Steps

*   **[Google Sheets Guide](/docs/guides/connections/google-sheets.md)**: Complete connection reference
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Advanced caching strategies
*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: When to use full refresh, incremental append, or incremental merge
*   **[Authentication](/docs/endpoints/authentication.md)**: Add API keys
*   **[Deployment](/docs/getting-started/deployment.md)**: Deploy to production

## Complete Code

All files for this example are available in the [flAPI Examples Repository](https://github.com/datazoode/flapi-examples/tree/main/google-sheets-catalog).

---

**Marketing Loves This:**

This pattern is a game-changer for teams where non-technical staff manage content. Marketing can update product descriptions, pricing, and availability without waiting for developers. The API provides a clean interface for developers while giving business users full control.

([🍪 Cookie Settings](#cookie-settings))

# Guided Agent Workflows with MCP Prompts

You can expose 50 MCP tools and still get poor answers from an AI agent — because the agent doesn't know in what **order** to call them, or how to summarise the results. A user asks "is this customer at risk?" and the agent makes one tool call when it should have made three.

**MCP prompts** are the fix. A prompt is a Mustache template authored by the API operator, published over MCP, and rendered on demand by the server. The agent fetches the rendered text via `prompts/get` and then follows the instructions inside it. The operator gets to script the workflow once; every agent gets the same playbook.

This recipe builds two prompts that demonstrate the pattern.

## What You'll Build

Two MCP prompts on top of an existing customer API:

1.  `customer_analysis` — given a customer id (and optionally a segment, analysis type, and time period), produces a structured analysis script that names the right tools and resources in the right order.
2.  `data_quality_check` — given a table, a column, and a threshold, produces a checklist for validating data freshness, null rates, and distribution.

Crucially, prompts **do not execute SQL**. They render text. The agent then chooses to call the tools and resources the prompt mentions.

## How Prompts Differ From Tools and Resources

The prompt is **a script for the agent**. flAPI's `prompts/get` handler renders Mustache placeholders against the supplied arguments and returns the result as a chat message — see `mcp-prompt:` handling in `src/mcp_route_handlers.cpp` and the response shape in [MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md#promptslist-and-promptsget).

## Prerequisites

*   An existing flAPI project with at least one `mcp-tool` declared (we'll reference `customer_lookup`)
*   An existing MCP resource for the schema is useful but optional — see [Schema as an MCP Resource](/docs/examples/mcp-resources.md)

## Project Layout

```
prompts-demo/├── flapi.yaml├── data/│   └── customers.parquet└── sqls/    ├── customer-common.yaml    ├── customer-lookup-tool.yaml        # existing mcp-tool    ├── customer-schema-resource.yaml    # existing mcp-resource    ├── customer-analysis-prompt.yaml    # NEW    └── data-quality-check-prompt.yaml   # NEW
```

Prompts live in their own YAML files alongside the rest of the endpoint config; flAPI picks them up by the presence of an `mcp-prompt:` block at the top.

## Step-by-Step

### 1\. The `customer_analysis` Prompt

**`sqls/customer-analysis-prompt.yaml`:**

```
mcp-prompt:  name: customer_analysis  description: Generate a structured customer analysis playbook for the agent  template: |    You are a customer data analyst. Follow this playbook precisely.    ## Analysis Request    {{#customer_id}}    - Customer ID: {{customer_id}}    {{/customer_id}}    {{#segment}}    - Segment focus: {{segment}}    {{/segment}}    {{#analysis_type}}    - Analysis type: {{analysis_type}}    {{/analysis_type}}    {{#time_period}}    - Time period: {{time_period}}    {{/time_period}}    ## Step 1 — Ground yourself    Call the `customer_schema` MCP resource (URI `flapi://customer_schema`)    to confirm field names and types before issuing any tool call.    ## Step 2 — Fetch the customer    Call the `customer_lookup` MCP tool with arguments:        { "id": "{{customer_id}}"{{#segment}}, "segment": "{{segment}}"{{/segment}} }    Confirm the returned `c_mktsegment` matches the segment focus    {{#segment}}({{segment}}){{/segment}}. If it does not, stop and report    the discrepancy to the user.    ## Step 3 — Profile and recommend    Using the row returned by `customer_lookup`:    1. Summarise the customer profile in one paragraph.    2. Flag any of these risk signals:       - `c_acctbal` below 0       - `c_address` empty or single line       - `c_phone` not matching the regional format    3. Recommend the next best action ({{#analysis_type}}{{analysis_type}}-style{{/analysis_type}}).    {{#include_schema}}    ## Step 4 — Append schema reference    Quote the relevant columns from `flapi://customer_schema` so the user    can see the data types behind your analysis.    {{/include_schema}}    Respond in Markdown.  arguments:    - customer_id    - segment    - analysis_type    - time_period    - include_schema
```

Key things to notice:

*   **Inline template only.** Unlike `mcp-tool` and `mcp-resource`, prompts use an inline `template:` string and **do not** support `template-source:` files. (See `CONFIG_REFERENCE.md §3.4`.)
*   **No `connection:` block.** Prompts never run SQL, so there is no database connection to declare.
*   **Mustache sections gate optional content.** `{{#segment}}…{{/segment}}` renders only when `segment` is supplied. This lets one prompt template cover several call shapes.
*   **The prompt names the tools.** `customer_lookup` and `flapi://customer_schema` are referenced by exact name so the agent knows which `tools/call` / `resources/read` to make next.

### 2\. The `data_quality_check` Prompt

A second prompt to show multi-argument substitution and a slightly different style.

**`sqls/data-quality-check-prompt.yaml`:**

```
mcp-prompt:  name: data_quality_check  description: Generate a data-quality checklist for a single column  template: |    You are a data quality engineer reviewing the column `{{column}}`    of table `{{table}}`.    Apply the following checks in order. For each check, call the    appropriate flAPI MCP tool and report PASS / FAIL / SKIP with    a one-line justification.    ## Checks    1. **Freshness** — confirm the most recent row is no older than       {{#max_age_hours}}{{max_age_hours}}{{/max_age_hours}}{{^max_age_hours}}24{{/max_age_hours}} hours.    2. **Null rate** — confirm that fewer than       {{#null_threshold_pct}}{{null_threshold_pct}}{{/null_threshold_pct}}{{^null_threshold_pct}}5{{/null_threshold_pct}}%       of rows have a NULL `{{column}}`.    3. **Distinct count** — confirm the column is not collapsed to a single       distinct value (which often indicates a broken ingest).    {{#check_distribution}}    4. **Distribution** — call the `column_histogram` tool and verify       the top bucket is no more than 80% of the total.    {{/check_distribution}}    ## Output    Return a Markdown table:    | Check | Result | Notes |    |-------|--------|-------|    End with a one-paragraph "Overall verdict" recommending one of:    ACCEPT, ACCEPT-WITH-WARNINGS, REJECT.  arguments:    - table    - column    - max_age_hours    - null_threshold_pct    - check_distribution
```

This prompt uses **inverted sections** — `{{^max_age_hours}}24{{/max_age_hours}}` — to inline a default value when the argument isn't supplied. That keeps the playbook self-contained even when the caller omits optional inputs.

## End-to-End JSON-RPC Walkthrough

### 1\. Initialize and discover

```
$ curl -sS -i -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -d '{    "jsonrpc": "2.0",    "id": 1,    "method": "initialize",    "params": {      "protocolVersion": "2025-11-25",      "clientInfo": {"name": "curl-demo", "version": "1.0.0"}    }  }'HTTP/1.1 200 OKMcp-Session-Id: 7a1c-...-9f88
```

```
$ SID=7a1c-...-9f88
```

### 2\. List prompts

```
$ curl -sS -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -H "Mcp-Session-Id: $SID" \  -d '{"jsonrpc":"2.0","id":2,"method":"prompts/list","params":{}}' | jq
```

```
{  "jsonrpc": "2.0",  "id": 2,  "result": {    "prompts": [      {        "name": "customer_analysis",        "description": "Generate a structured customer analysis playbook for the agent",        "arguments": [          { "name": "customer_id" },          { "name": "segment" },          { "name": "analysis_type" },          { "name": "time_period" },          { "name": "include_schema" }        ]      },      {        "name": "data_quality_check",        "description": "Generate a data-quality checklist for a single column",        "arguments": [          { "name": "table" },          { "name": "column" },          { "name": "max_age_hours" },          { "name": "null_threshold_pct" },          { "name": "check_distribution" }        ]      }    ]  }}
```

### 3\. Render `customer_analysis`

```
$ curl -sS -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -H "Mcp-Session-Id: $SID" \  -d '{    "jsonrpc": "2.0",    "id": 3,    "method": "prompts/get",    "params": {      "name": "customer_analysis",      "arguments": {        "customer_id": "12345",        "segment": "AUTOMOBILE",        "analysis_type": "churn-risk",        "include_schema": true      }    }  }' | jq
```

```
{  "jsonrpc": "2.0",  "id": 3,  "result": {    "description": "Generate a structured customer analysis playbook for the agent",    "messages": [      {        "role": "user",        "content": {          "type": "text",          "text": "You are a customer data analyst. Follow this playbook precisely.\n\n## Analysis Request\n- Customer ID: 12345\n- Segment focus: AUTOMOBILE\n- Analysis type: churn-risk\n\n## Step 1 — Ground yourself\nCall the `customer_schema` MCP resource ..."        }      }    ]  }}
```

Notice three things:

*   The `time_period` argument was omitted, so the entire `{{#time_period}}…{{/time_period}}` section disappeared from the output.
*   `include_schema: true` enabled the "Step 4 — Append schema reference" section.
*   The result is a chat message (`role: "user"`), not raw text. flAPI wraps the rendered template in a single user-role message so that the client can drop it straight into an LLM conversation.

### 4\. Render `data_quality_check` with defaults

```
$ curl -sS -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -H "Mcp-Session-Id: $SID" \  -d '{    "jsonrpc": "2.0",    "id": 4,    "method": "prompts/get",    "params": {      "name": "data_quality_check",      "arguments": {        "table": "customers",        "column": "c_acctbal"      }    }  }' | jq -r '.result.messages[0].content.text'
```

```
You are a data quality engineer reviewing the column `c_acctbal`of table `customers`....1. Freshness — confirm the most recent row is no older than 24 hours.2. Null rate — confirm that fewer than 5% of rows have a NULL `c_acctbal`....
```

The inverted sections injected the `24` and `5` defaults because the agent didn't supply `max_age_hours` or `null_threshold_pct`. The optional `check_distribution` step was omitted entirely.

## Prompts Don't Run SQL — Why That Matters

This is the single biggest misconception about MCP prompts:

> A `prompts/get` call **renders text**. It does not query the database, it does not call other tools, and it does not authenticate against your `auth` block.

The server reads the `mcp-prompt.template` string, substitutes Mustache placeholders against the supplied `arguments`, and returns the rendered chat message. The agent reads the response and decides what to do next — usually a `tools/call` to one of the tools mentioned in the rendered text.

A consequence: **placeholders are substituted as-is**. If a malicious caller supplies `"customer_id": "12345; DROP TABLE customers"`, that string ends up inside the rendered prompt — but it is **text for the agent to read**, not SQL passed to a database. The agent is expected to call `customer_lookup` with that value, at which point flAPI's `request` validators on the tool will reject it (see [Validation](/docs/endpoints/validation.md)).

If you want a prompt to consume live data, design it so the prompt instructs the agent to call a tool, and put the validation on the tool.

## Mustache Cheat-Sheet

| Syntax | Meaning |
| --- | --- |
| `{{customer_id}}` | HTML-escape and substitute the value |
| `{{{customer_id}}}` | Substitute the value without escaping (rare in prompts; common in SQL templates) |
| `{{#segment}}…{{/segment}}` | Render the section only if `segment` is present and truthy |
| `{{^segment}}…{{/segment}}` | Render the section only if `segment` is **absent** or falsey |
| `{{! comment }}` | Ignored at render time |

Arrays are also supported (`{{#items}}{{.}}{{/items}}`), but prompts rarely need them — most playbooks use scalar arguments only. For the full grammar see the upstream Mustache spec referenced by flAPI's template engine.

## Tying It Together

The most productive pattern is:

1.  **One MCP resource** for the schema (`customer_schema`) — provides shared context.
2.  **One MCP tool** for the data (`customer_lookup`) — does the work.
3.  **One MCP prompt** for each common workflow (`customer_analysis`, `data_quality_check`) — chains the previous two.

A single agent session might do `prompts/get` → `resources/read` → `tools/call` → final answer, with the operator's prompt template doing the heavy lifting of orchestration.

## See Also

*   **[MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md#promptslist-and-promptsget)** — the wire shape of `prompts/list` and `prompts/get`
*   **[MCP Overview](/docs/ai-integration/mcp-overview.md)** — when to choose a prompt over a tool or resource
*   **[Schema as an MCP Resource](/docs/examples/mcp-resources.md)** — the companion recipe for the `customer_schema` resource this prompt references
*   **[Validation](/docs/endpoints/validation.md)** — the validator framework that protects tools when a prompt's instructions trigger a tool call
*   **[Claude Integration](/docs/ai-integration/claude-integration.md)** — wiring Claude Desktop to discover and render these prompts automatically

([🍪 Cookie Settings](#cookie-settings))

# Expose Schema and Docs as MCP Resources

When you wire an AI agent to flAPI, it can only call tools intelligently if it knows what fields exist, what they mean, and what shape the data is in. Telling the agent "call `customer_lookup`" is not enough — it also needs to know that `c_mktsegment` accepts one of five enum values, that `c_acctbal` is a `DOUBLE`, and that the table is sourced from TPC-H.

This is what **MCP resources** are for: read-only, semi-static payloads that an agent fetches once (or on demand) to ground itself before it starts invoking tools.

## What You'll Build

Two MCP resources served by a single flAPI server:

1.  `customer_schema` — a JSON description of the customer table's columns, types, and descriptions.
2.  `customer_data_dictionary` — a Markdown reference that explains business meaning, segment definitions, and known gotchas.

An MCP-compatible client (Claude Desktop, Claude Code, a custom agent) will discover both via `resources/list`, fetch them via `resources/read`, and use the information to issue better tool calls.

## How It Works

Resources are addressed by a flat URI scheme — `flapi://<resource_name>` — and returned as either `text` or base64-encoded `blob` content. There are no path components; the server resolves the name directly against the `mcp-resource.name` declared in each YAML file (see `src/mcp_route_handlers.cpp`).

## Prerequisites

*   flAPI installed ([Quickstart](/docs/getting-started/quickstart.md))
*   A Parquet file of customer data (we use the TPC-H sample)

## Project Layout

```
schema-resources/├── flapi.yaml├── data/│   └── customers.parquet└── sqls/    ├── customer-common.yaml    ├── customer-schema-resource.yaml    ├── customer-schema.sql    ├── customer-data-dictionary.yaml    └── customer-data-dictionary.sql
```

We are not using DuckLake or caching here — resources are typically tiny and either constant or generated from `information_schema`.

## Step-by-Step

### 1\. Main Configuration

**`flapi.yaml`:**

```
project-name: schema-resourcesproject-description: Expose customer schema and dictionary as MCP resourcestemplate:  path: './sqls'connections:  customers-parquet:    properties:      path: './data/customers.parquet'duckdb:  access_mode: READ_ONLYmcp:  enabled: true
```

Setting `mcp.enabled: true` is optional — flAPI auto-enables MCP whenever any endpoint declares an `mcp-tool`, `mcp-resource`, or `mcp-prompt` block — but it makes the intent explicit.

### 2\. Shared Connection

**`sqls/customer-common.yaml`:**

```
# Reusable connection reference used by every resource and toolconnection:  - customers-parquet
```

This mirrors the include pattern from the upstream `examples/sqls/customers/customer-common.yaml`.

### 3\. The Schema Resource

**`sqls/customer-schema-resource.yaml`:**

```
mcp-resource:  name: customer_schema  description: Customer table schema, column types, and field descriptions  mime-type: application/json# Resources don't accept request parameters — they're read by URI alonetemplate-source: customer-schema.sql{{include:connection from customer-common.yaml}}# Resources change rarely; light rate limit is plentyrate-limit:  enabled: true  max: 10  interval: 60
```

This is the canonical `mcp-resource` shape: `name`, `description`, and `mime-type` at the top, a `template-source` pointing at the SQL that generates the body, and an included `connection`. The schema accepts no parameters, so we omit the `request:` block entirely.

**`sqls/customer-schema.sql`:**

```
-- Returns one row whose only column is a JSON document describing-- the customer table. The MCP layer serialises this row as the-- resource body.SELECT json_object(  'table_name', 'customers',  'source',     'TPC-H customer.parquet',  'description', 'Customer master records used for lookups and analytics',  'columns', json_array(    json_object('name', 'c_custkey',    'type', 'INTEGER',  'description', 'Unique customer key (primary key)'),    json_object('name', 'c_name',       'type', 'VARCHAR',  'description', 'Customer display name'),    json_object('name', 'c_address',    'type', 'VARCHAR',  'description', 'Postal address (free-form)'),    json_object('name', 'c_nationkey',  'type', 'INTEGER',  'description', 'FK into nation dimension'),    json_object('name', 'c_phone',      'type', 'VARCHAR',  'description', 'Phone number (free-form formatting)'),    json_object('name', 'c_acctbal',    'type', 'DOUBLE',   'description', 'Account balance in account currency'),    json_object('name', 'c_mktsegment', 'type', 'VARCHAR',  'description', 'One of: AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY'),    json_object('name', 'c_comment',    'type', 'VARCHAR',  'description', 'Free-form notes')  )) AS schema_definition;
```

The result has a single column and a single row. flAPI's MCP layer takes that row and emits it inside the `contents[0].text` field of the `resources/read` response.

### 4\. The Data Dictionary Resource

The second resource shows that a resource is just "text addressed by a URI" — it doesn't have to be JSON.

**`sqls/customer-data-dictionary.yaml`:**

```
mcp-resource:  name: customer_data_dictionary  description: Business glossary and segment definitions for the customer dataset  mime-type: text/markdowntemplate-source: customer-data-dictionary.sql{{include:connection from customer-common.yaml}}rate-limit:  enabled: true  max: 10  interval: 60
```

**`sqls/customer-data-dictionary.sql`:**

```
-- Inline Markdown reference. The body is just a string column.SELECT '# Customer Data Dictionary## Market Segments (`c_mktsegment`)| Code        | Meaning                              ||-------------|--------------------------------------|| AUTOMOBILE  | Auto OEMs and dealers                || BUILDING    | Construction, real estate developers || FURNITURE   | Home and office furnishing retail    || HOUSEHOLD   | Consumer goods, supermarkets         || MACHINERY   | Industrial machinery customers       |## Known Gotchas- `c_acctbal` may be negative (post-paid customers).- `c_phone` is not normalised — do not match by raw string.- `c_address` includes line breaks; strip them before display.' AS markdown_body;
```

The MIME type is `text/markdown`, so a client that knows how to render Markdown can do so directly. An LLM reading the body as plain text also gets the same information.

### Important: URIs are flat

flAPI's resource URI scheme has no path component. Both resources above are accessible only at:

*   `flapi://customer_schema`
*   `flapi://customer_data_dictionary`

There is **no** `flapi://docs/customer` style URI — the substring after `flapi://` is matched literally against an `mcp-resource.name` (see `src/mcp_route_handlers.cpp:955`). If you want to parameterise the content, expose an **MCP tool** instead — tools accept structured `arguments`, resources do not.

## Running the Server

```
$ ./flapi -c flapi.yamlok  Loaded 2 MCP resourcesok  MCP server ready at POST /mcp/jsonrpcok  Server listening on :8080
```

## End-to-End JSON-RPC Walkthrough

The MCP server speaks JSON-RPC 2.0 over `POST /mcp/jsonrpc`. See [MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md) for the full transport spec.

### 1\. Initialize the session

```
$ curl -sS -i -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -d '{    "jsonrpc": "2.0",    "id": 1,    "method": "initialize",    "params": {      "protocolVersion": "2025-11-25",      "clientInfo": {"name": "curl-demo", "version": "1.0.0"}    }  }'HTTP/1.1 200 OKMcp-Session-Id: 7a1c-...-9f88Content-Type: application/json{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-11-25","capabilities":{...},"serverInfo":{"name":"flapi-mcp-server","version":"0.3.0"}}}
```

Capture the session id and reuse it on every subsequent request:

```
$ SID=7a1c-...-9f88
```

### 2\. Discover resources

```
$ curl -sS -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -H "Mcp-Session-Id: $SID" \  -d '{"jsonrpc":"2.0","id":2,"method":"resources/list","params":{}}' | jq
```

```
{  "jsonrpc": "2.0",  "id": 2,  "result": {    "resources": [      {        "name": "customer_schema",        "description": "Customer table schema, column types, and field descriptions",        "mimeType": "application/json",        "uri": "flapi://customer_schema"      },      {        "name": "customer_data_dictionary",        "description": "Business glossary and segment definitions for the customer dataset",        "mimeType": "text/markdown",        "uri": "flapi://customer_data_dictionary"      }    ]  }}
```

### 3\. Read the schema

```
$ curl -sS -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -H "Mcp-Session-Id: $SID" \  -d '{    "jsonrpc": "2.0",    "id": 3,    "method": "resources/read",    "params": { "uri": "flapi://customer_schema" }  }' | jq
```

```
{  "jsonrpc": "2.0",  "id": 3,  "result": {    "contents": [      {        "uri": "flapi://customer_schema",        "mimeType": "application/json",        "text": "{\"table_name\":\"customers\",\"source\":\"TPC-H customer.parquet\",\"columns\":[...]}"      }    ]  }}
```

The `text` field is a JSON string (note the escaped quotes) — clients are expected to JSON-parse it themselves before display.

### 4\. Read the data dictionary

```
$ curl -sS -X POST http://localhost:8080/mcp/jsonrpc \  -H "Content-Type: application/json" \  -H "Mcp-Session-Id: $SID" \  -d '{    "jsonrpc": "2.0",    "id": 4,    "method": "resources/read",    "params": { "uri": "flapi://customer_data_dictionary" }  }' | jq -r '.result.contents[0].text'
```

```
# Customer Data Dictionary## Market Segments (`c_mktsegment`)...
```

The agent now has both a machine-readable schema and a human-readable glossary. It can use the schema to validate tool arguments before calling `customer_lookup`, and the dictionary to write better natural-language answers.

## When Resources Beat Tools

Resources and tools overlap — both can return JSON, both are described to the agent at startup. Reach for a resource when the answer is essentially **read-only reference material**:

| Choose a resource when… | Choose a tool when… |
| --- | --- |
| The content is static or rarely changes | The content depends on user input |
| The agent should read it once and cache the result | Every call needs fresh data |
| There are no meaningful arguments | The call accepts filters, IDs, dates, etc. |
| You want it to appear in `resources/list` for context loading | You want it in `tools/list` for invocation |

A useful rule of thumb: **if the agent would ask "what's available?" before calling anything, that "what's available?" answer belongs in a resource.** Catalogues, schemas, enum definitions, policy documents, sample payloads, and onboarding instructions all fit naturally.

For everything else — anything that filters, paginates, or writes — define an `mcp-tool` instead. See [MCP Overview](/docs/ai-integration/mcp-overview.md) for the side-by-side YAML shapes.

## Securing Sensitive Resources

By default, resources are unauthenticated — the same as `resources/list`. If a resource exposes sensitive metadata (internal column names, regulated schemas), attach the same `auth` and `rate-limit` blocks you would on a tool:

```
mcp-resource:  name: pii_schema  description: PII column inventory (restricted)  mime-type: application/jsontemplate-source: pii-schema.sql{{include:connection from customer-common.yaml}}{{include:auth from customer-common.yaml}}rate-limit:  enabled: true  max: 5  interval: 60
```

The `auth` block resolves to the same basic/JWT/bearer schemes documented in [Authentication](/docs/endpoints/authentication.md).

## See Also

*   **[MCP Protocol Reference](/docs/ai-integration/mcp-protocol.md)** — the JSON-RPC envelope, session lifecycle, error codes
*   **[MCP Overview](/docs/ai-integration/mcp-overview.md)** — choosing between tools, resources, and prompts
*   **[YAML Syntax & Structure](/docs/endpoints/yaml-syntax.md)** — full endpoint syntax including `{{include:...}}`
*   **[Guided Agent Workflows with MCP Prompts](/docs/examples/mcp-prompts.md)** — pair resources with prompts for context-rich agents
*   **[Claude Integration](/docs/ai-integration/claude-integration.md)** — wiring Claude Desktop to consume resources

([🍪 Cookie Settings](#cookie-settings))

# Multi-Environment Deployment from S3

You want one flAPI container image, three running environments — `dev`, `staging`, `prod` — each picking up its own `flapi.yaml` and SQL templates from S3. When somebody merges to `main`, the prod templates in S3 get refreshed; the running pods pull the new config on their next reload. No image rebuild, no rolling deployment for a configuration-only change.

This recipe wires that up end-to-end against AWS S3. The same pattern works for GCS and Azure Blob — only the URI scheme and credentials change.

## Architecture

```
                      git push to main                              |                              v                    +-------------------+                    |   GitHub Action   |   syncs config/ -> s3://my-flapi-config/prod/                    +-------------------+                              |                              v                +---------------------------+                |   s3://my-flapi-config/   |                |     dev/                  |                |     staging/              |                |     prod/                 |                +---------------------------+                      ^         ^         ^                      |         |         |              flapi --config s3://...   (per environment)                      ^         ^         ^                      |         |         |                    [dev]   [staging]   [prod]   pods
```

The flAPI binary loads YAML and SQL files through DuckDB's virtual filesystem (VFS), so any URI DuckDB can read, flAPI can read.

## S3 Layout

```
s3://my-flapi-config/  dev/    flapi.yaml    sqls/      customers.yaml      customers.sql  staging/    flapi.yaml    sqls/      customers.yaml      customers.sql  prod/    flapi.yaml    sqls/      customers.yaml      customers.sql
```

The split is fully physical — each environment has its own folder. There is no `{{env.STAGE}}` magic; the deployment knows which folder to load by passing the right `--config` URI.

## Deployment Invocations

The same image, three different startup commands:

```
# devflapi --config s3://my-flapi-config/dev/flapi.yaml# stagingflapi --config s3://my-flapi-config/staging/flapi.yaml# prodflapi --config s3://my-flapi-config/prod/flapi.yaml
```

If you also expose the Config Service REST API for hot reloads, add `--config-service`:

```
flapi --config s3://my-flapi-config/prod/flapi.yaml \      --config-service \      --config-service-token "$CONFIG_SERVICE_TOKEN"
```

See the [Config Service REST API](/docs/tools/config-service-api.md) for the reload endpoints.

## The flapi.yaml in S3

One example for prod. `template.path` is set to a sibling prefix in the **same** bucket so that endpoint YAMLs and SQL files don't need full URIs.

```
# s3://my-flapi-config/prod/flapi.yamlproject-name: products-api-prodproject-description: Production deployment, config from S3template:  path: s3://my-flapi-config/prod/sqls/  environment-whitelist:    - '^AWS_.*'    - '^FLAPI_.*'connections:  products-data:    init: |      INSTALL httpfs;      LOAD httpfs;      SET s3_region='us-east-1';    properties:      base_path: s3://my-data/parquet/      path: s3://my-data/parquet/products.parquetduckdb:  access_mode: READ_WRITE  threads: 8ducklake:  enabled: true  alias: cache  metadata-path: ./data/cache.ducklake  data-path: ./data/cache  retention:    keep-last-snapshots: 30    max-snapshot-age: 14d  scheduler:    enabled: true    scan-interval: 5mstorage:  cache:    enabled: true    ttl: 300    max_size: 50MB
```

The dev and staging variants are byte-identical except for the bucket prefix, connection paths, and retention numbers.

## Required Environment Variables

flAPI talks to S3 through DuckDB's `httpfs` extension, which reads credentials from the standard AWS environment:

| Variable | Required? | Notes |
| --- | --- | --- |
| `AWS_ACCESS_KEY_ID` | yes (off-AWS) | Provided automatically on EC2/ECS/EKS via IAM role |
| `AWS_SECRET_ACCESS_KEY` | yes (off-AWS) | Same as above |
| `AWS_REGION` | yes | e.g. `us-east-1` |
| `AWS_SESSION_TOKEN` | optional | For STS / assumed-role credentials |
| `AWS_ENDPOINT_URL` | optional | For S3-compatible storage (MinIO, LocalStack) |

When the pod runs inside AWS with an attached IAM role, **no environment variables are needed** — the role is picked up by the AWS SDK chain automatically.

## Scheme Whitelisting

By default flAPI's path validator only allows two URI schemes:

*   `file://` (and plain local paths)
*   `https://`

This is enforced by `PathValidator::Config::allowed_schemes` in `src/include/path_validator.hpp`, which is the gate that every config and template path passes through before it's resolved.

To use `s3://` for `--config` you have to opt in by letting flAPI know that S3 is a real, configured backend. The pattern is to declare a connection whose `init:` block installs and loads the `httpfs` extension:

```
connections:  cloud-data:    init: |      INSTALL httpfs;      LOAD httpfs;      SET s3_region='us-east-1';    properties:      base_path: s3://my-flapi-config/prod/
```

Once `httpfs` is loaded and an S3-backed connection exists, `s3://` becomes available for `--config`, `template.path`, and any `template-source`. The same applies to `gs://` and `az://` — install/load the right extension in a connection's `init` block.

If you skip this and pass `--config s3://...` cold, flAPI will reject the path with `URL scheme not allowed: s3` from `PathValidator::ValidateRemotePath`.

## IAM Permissions

The pod's IAM role (or static credentials) needs read access to the bucket prefix. Minimum policy:

```
{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "ReadFlapiConfig",      "Effect": "Allow",      "Action": [        "s3:GetObject"      ],      "Resource": "arn:aws:s3:::my-flapi-config/*"    },    {      "Sid": "ListFlapiConfigBucket",      "Effect": "Allow",      "Action": [        "s3:ListBucket"      ],      "Resource": "arn:aws:s3:::my-flapi-config"    }  ]}
```

`s3:GetObject` is needed for every YAML and SQL file read. `s3:ListBucket` is needed for `template.path` directory enumeration. Scope the resource ARN to a single environment prefix (e.g. `my-flapi-config/prod/*`) for stricter isolation.

If your data sources also live on S3, add a second statement covering the data bucket.

## GitOps Workflow

The idea: the canonical configuration lives in git. A merge to `main` syncs the relevant subfolder to S3, and the running pods pick the change up on their next config reload.

Repository layout:

```
my-flapi-configs/  dev/    flapi.yaml    sqls/...  staging/    flapi.yaml    sqls/...  prod/    flapi.yaml    sqls/...  .github/    workflows/      sync-config.yml
```

### GitHub Actions workflow

```
# .github/workflows/sync-config.ymlname: Sync flAPI config to S3on:  push:    branches: [main]    paths:      - 'dev/**'      - 'staging/**'      - 'prod/**'permissions:  id-token: write  contents: readjobs:  sync:    runs-on: ubuntu-latest    strategy:      matrix:        env: [dev, staging, prod]    steps:      - uses: actions/checkout@v4      - name: Configure AWS credentials        uses: aws-actions/configure-aws-credentials@v4        with:          role-to-assume: arn:aws:iam::123456789012:role/flapi-config-sync          aws-region: us-east-1      - name: Sync ${{ matrix.env }} to S3        run: |          aws s3 sync ./${{ matrix.env }}/ \            s3://my-flapi-config/${{ matrix.env }}/ \            --delete \            --exclude '.git*'
```

Optional add-ons for production hygiene:

*   Run a `flapi --validate-config s3://my-flapi-config/staging/flapi.yaml` step before syncing prod, to catch broken YAML.
*   After the sync, hit each pod's per-endpoint reload route `POST /api/v1/_config/endpoints/{slug}/reload` (or `POST /api/v1/_config/schema/refresh` for schema-level changes) via the [Config Service API](/docs/tools/config-service-api.md) so changes take effect within seconds rather than at the next scheduled reload.

## Hot Reload

flAPI does not poll S3 on every request. Remote files are cached in an LRU per the `storage.cache` block (default TTL 300s) — that's the [VFS cache documented in the Cloud Storage guide](/docs/guides/cloud-storage.md). Two ways to push config out faster:

1.  **Lower the TTL** on `storage.cache.ttl` if you want passive reload, at the cost of more S3 reads.
2.  **Call the Config Service** for active reload. With `--config-service` enabled, the running pod exposes per-endpoint reload (`POST /api/v1/_config/endpoints/{slug}/reload`) and schema refresh (`POST /api/v1/_config/schema/refresh`) routes that re-read from S3 immediately. See the [Config Service REST API](/docs/tools/config-service-api.md).

The CI workflow above can issue an authenticated POST to the prod pods at the end of the sync step to make the rollout effectively instantaneous.

## Kubernetes Deployment Snippet

A trimmed prod Deployment. The same manifest deployed three times (or via Kustomize/Helm) covers all environments — only the `FLAPI_CONFIG_S3_URL` ConfigMap key changes.

```
apiVersion: v1kind: ConfigMapmetadata:  name: flapi-env-prod  namespace: flapidata:  FLAPI_CONFIG_S3_URL: s3://my-flapi-config/prod/flapi.yaml  AWS_REGION: us-east-1---apiVersion: apps/v1kind: Deploymentmetadata:  name: flapi-prod  namespace: flapispec:  replicas: 3  selector:    matchLabels:      app: flapi      env: prod  template:    metadata:      labels:        app: flapi        env: prod    spec:      serviceAccountName: flapi-prod  # IRSA-bound to the IAM role above      containers:        - name: flapi          image: ghcr.io/datazoode/flapi:latest          args:            - --config            - $(FLAPI_CONFIG_S3_URL)            - --config-service          envFrom:            - configMapRef:                name: flapi-env-prod          env:            - name: CONFIG_SERVICE_TOKEN              valueFrom:                secretKeyRef:                  name: flapi-prod-secrets                  key: config-service-token          ports:            - containerPort: 8080              name: http          readinessProbe:            httpGet:              path: /api/v1/_config/health              port: http            initialDelaySeconds: 5            periodSeconds: 10          resources:            requests:              cpu: 200m              memory: 512Mi            limits:              cpu: 2              memory: 4Gi
```

Notes:

*   `serviceAccountName: flapi-prod` is bound to the IAM role with the policy from the previous section via IRSA (IAM Roles for Service Accounts). No `AWS_ACCESS_KEY_ID` is set on the pod.
*   `args` references the env var with shell-style `$(VAR)` substitution — Kubernetes resolves it before exec.
*   The readiness probe hits the Config Service health endpoint, which reports VFS status (the bucket has to be reachable for the pod to be ready).
*   Three replicas are intentional: a config rollout via the GitHub Action plus a fan-out `POST /api/v1/_config/endpoints/{slug}/reload` (per endpoint) propagates to all three pods.

## Testing the Setup Locally

Before deploying, smoke-test the same invocation against MinIO or LocalStack:

```
# Start LocalStack with S3docker run -d --name localstack -p 4566:4566 localstack/localstack# Set the S3-compatible endpointexport AWS_ACCESS_KEY_ID=testexport AWS_SECRET_ACCESS_KEY=testexport AWS_REGION=us-east-1export AWS_ENDPOINT_URL=http://localhost:4566# Upload one envaws --endpoint-url $AWS_ENDPOINT_URL s3 mb s3://my-flapi-configaws --endpoint-url $AWS_ENDPOINT_URL s3 sync ./prod/ s3://my-flapi-config/prod/# Run flAPI against itflapi --config s3://my-flapi-config/prod/flapi.yaml
```

If you see `URL scheme not allowed: s3`, your `flapi.yaml` is missing a connection that installs `httpfs` — see ([Scheme Whitelisting](#scheme-whitelisting)).

## Common Pitfalls

*   **Forgetting `s3:ListBucket`.** Reading individual files works, but `template.path: s3://.../sqls/` directory enumeration fails silently.
*   **Mixing buckets across environments.** If `prod` accidentally points `template.path` at the dev bucket, the difference will be invisible until a prod-only template gets edited. Keep prefixes physically separate and lock down IAM per environment.
*   **Region drift.** Pods running in `eu-west-1` reading from a `us-east-1` bucket will work but with measurable latency on every cache miss. Co-locate the config bucket with the running cluster, or lean on a longer `storage.cache.ttl`.
*   **Forgetting to call reload.** Pods will eventually pick up new config when the LRU TTL elapses, but if you need a guaranteed rollout, finish your CI pipeline with a Config Service refresh call.

## Next Steps

*   [Cloud Storage and VFS](/docs/guides/cloud-storage.md) — full reference for all supported URI schemes (S3, GCS, Azure, HTTPS) and credential sources.
*   [Config Service REST API](/docs/tools/config-service-api.md) — endpoints for runtime reloads, manual cache refresh, and health checks.
*   [Deployment guide](/docs/getting-started/deployment.md) — broader deployment topologies including Docker, ConfigMaps, Cloud Run, and Lambda.

([🍪 Cookie Settings](#cookie-settings))

# Multi-Tenant SaaS API with OIDC

You run a B2B SaaS company. Customer A is a bank. Customer B is a hospital chain. Customer C is a 12-person startup. They all hit the same flAPI deployment, but every API response must be perfectly scoped to the calling tenant.

You do **not** want to:

*   Store customer passwords.
*   Build a federation layer on top of every customer's IdP.
*   Issue and rotate per-tenant API keys.

You **do** want to:

*   Let each customer authenticate against their own Microsoft Entra ID (Azure AD) tenant.
*   Verify tokens locally using the IdP's JWKS — no callback to the IdP per request.
*   Use the JWT's `tenant_id` and `roles` claims directly in the SQL template for row-level security.

flAPI's OIDC support, implemented in `oidc_auth_handler.cpp` and configured under `auth.oidc.*`, is built exactly for this.

## Token Flow

```
┌────────────┐        ┌───────────────────────┐        ┌──────────┐│   Client   │        │  Microsoft Entra ID   │        │  flAPI   ││  (browser, │        │  login.microsoftonline│        │  /customers│   CLI, app)│        │       .com/{tenant}   │        │          │└─────┬──────┘        └──────────┬────────────┘        └────┬─────┘      │                          │                          │      │ 1. Sign in (OAuth2)      │                          │      ├─────────────────────────>│                          │      │                          │                          │      │ 2. Access token (JWT)    │                          │      │<─────────────────────────┤                          │      │                          │                          │      │ 3. GET /customers/                                  │      │    Authorization: Bearer eyJ...                     │      ├────────────────────────────────────────────────────>│      │                          │                          │      │                          │ 4. Fetch JWKS (cached)   │      │                          │<─────────────────────────┤      │                          │                          │      │                          │ 5. JWKS document         │      │                          ├─────────────────────────>│      │                          │                          │      │                          │     6. Verify RS256      │      │                          │        signature locally │      │                          │     7. Check iss / aud / │      │                          │        exp + clock-skew  │      │                          │     8. Map claims:       │      │                          │        preferred_username│      │                          │        roles, tenant_id  │      │                          │                          │      │ 9. 200 OK + tenant-scoped JSON                      │      │<────────────────────────────────────────────────────┤
```

JWKS is fetched the first time a token with a new `kid` arrives, then cached for `jwks-cache-hours` (default 24). Every subsequent request validates fully offline.

## Configuration

```
# flapi.yamlproject-name: customer-data-saasproject-description: Multi-tenant customer data API with Entra ID OIDCtemplate:  path: './sqls'  environment-whitelist:    - '^MS_.*'    - '^DB_.*'connections:  warehouse:    init: |      INSTALL postgres;      LOAD postgres;      ATTACH 'host={{env.DB_HOST}} dbname={{env.DB_NAME}} user={{env.DB_USER}} password={{env.DB_PASSWORD}}'        AS warehouse (TYPE postgres, READ_ONLY);# Global OIDC: applies to every endpoint unless an endpoint overrides it.auth:  enabled: true  type: oidc  oidc:    provider-type: microsoft    issuer-url: 'https://login.microsoftonline.com/{tenant}/v2.0'    client-id: '{{env.MS_CLIENT_ID}}'    allowed-audiences:      - 'api://flapi-saas'    scopes: [openid, profile, email]    username-claim: preferred_username    email-claim: email    roles-claim: roles    role-claim-path: roles    jwks-cache-hours: 12    clock-skew-seconds: 30enforce-https:  enabled: true
```

Every key above is parsed in `config_manager.cpp::parseOIDCConfigNode`. Anything not in that function is silently ignored by flAPI, so the snippet stays narrow on purpose.

### Why these particular values

*   **`provider-type: microsoft`** — applies the `microsoft` preset from `oidc_provider_presets.cpp`, which validates that `issuer-url` contains a `{tenant}` placeholder.
*   **`issuer-url: '.../{tenant}/v2.0'`** — Microsoft _requires_ the literal `{tenant}` placeholder in the issuer. The preset fails validation without it.
*   **`allowed-audiences: ['api://flapi-saas']`** — tokens not minted for your app are rejected by `OIDCAuthHandler::validateAudience`.
*   **`role-claim-path: roles`** — for Entra ID app roles, the claim is flat. If you ever switch to a provider that nests roles (Keycloak's `realm_access.roles`), only this one key changes.
*   **`jwks-cache-hours: 12`** — half the default. Tighter cache means faster recovery if Entra rotates a signing key.
*   **`clock-skew-seconds: 30`** — production servers should be NTP-synced, so the 300s default is generous. 30s catches genuinely expired tokens sooner.

## The Endpoint

```
# sqls/customers/customers.yamlurl-path: /customers/method: GETconnection:  - warehousetemplate-source: customers.sqlrequest:  - field-name: segment    field-in: query    description: Filter customers by segment    required: false  - field-name: limit    field-in: query    description: Maximum rows to return    required: false
```

No `auth:` block here — the endpoint inherits the global OIDC config.

## The SQL Template

`auth.username`, `auth.email`, and `auth.roles` are populated by the OIDC middleware after token validation. **`auth.roles` is a comma-joined string**, not a Mustache section — match it with `LIKE`.

```
-- sqls/customers/customers.sqlWITH caller AS (  SELECT    '{{{auth.username}}}'      AS user_login,    '{{{auth.email}}}'         AS user_email,    '{{auth.roles}}'           AS roles_csv)SELECT    c.customer_id,    c.tenant_id,    c.company_name,    c.segment,    c.countryFROM warehouse.public.customers cCROSS JOIN callerWHERE 1=1  -- Tenant scoping: derive tenant from the email domain (or a custom claim).  AND (       caller.roles_csv LIKE '%platform_admin%'   -- our own ops team: full read    OR c.tenant_email_domain = split_part(caller.user_email, '@', 2)  )  -- Within a tenant, only tenant_admin sees inactive customers.  AND (       caller.roles_csv LIKE '%tenant_admin%'    OR c.is_active = true  )  {{#params.segment}}  AND c.segment = '{{{params.segment}}}'  {{/params.segment}}ORDER BY c.company_name{{#params.limit}}LIMIT {{{params.limit}}}{{/params.limit}}
```

Two layers of authorization, both expressed in SQL:

1.  **Tenant boundary**: a user only sees rows whose `tenant_email_domain` matches the domain of their authenticated email. Platform admins (`platform_admin` role on your own Entra tenant) bypass this.
2.  **In-tenant role**: only `tenant_admin` sees deactivated customers within their tenant.

If a richer claim is available (for example, a custom `tenant_id` app role claim emitted by Entra), prefer matching on that instead of the email domain — it's harder to spoof and easier to audit.

## Other Provider Presets

flAPI ships seven preset values for `provider-type`. Every preset name below is enumerated in `oidc_provider_presets.cpp::applyPreset`.

| `provider-type` | One-liner |
| --- | --- |
| `google` | Google Workspace SSO. Default `username-claim: email`, issuer `https://accounts.google.com`. |
| `microsoft` | Microsoft Entra ID / Azure AD. Issuer `https://login.microsoftonline.com/{tenant}/v2.0` — `{tenant}` placeholder is required. |
| `keycloak` | Self-hosted Keycloak. Issuer `https://<host>/realms/{realm}`. Roles nest under `realm_access.roles` by default. |
| `auth0` | Auth0 SaaS. Issuer `https://{domain}.auth0.com`. Custom roles claim typically lives at a namespaced URL. |
| `okta` | Okta Workforce / Customer Identity. Issuer `https://{domain}.okta.com/oauth2/default`. |
| `github` | GitHub OAuth 2.0 (not full OIDC — token validation requires custom handling). |
| `generic` | Any spec-compliant OIDC IdP. You must provide `issuer-url` explicitly. |

Switching providers is usually just swapping `provider-type` and the issuer URL — claim mappings come from the preset.

## Per-Endpoint Override

Sometimes you want one endpoint open to a different audience. `auth` can be overridden per endpoint:

```
# sqls/health/health.yamlurl-path: /health/method: GETconnection:  - warehousetemplate-source: health.sqlauth:  enabled: false
```

A second OIDC pool — for example, a partner integration that authenticates against a different Entra tenant — is configured by repeating the full `auth.oidc` block under the endpoint. The endpoint-level block fully replaces the global one (see `parseEndpointAuth` in `config_manager.cpp`).

## Test It

### 1\. Get a token from Entra ID

For interactive testing, use the Azure CLI:

```
TENANT_ID=00000000-0000-0000-0000-000000000000CLIENT_ID=11111111-1111-1111-1111-111111111111TOKEN=$(az account get-access-token \  --tenant "$TENANT_ID" \  --resource "api://flapi-saas" \  --query accessToken -o tsv)
```

For service-to-service flows, request a token with the `client_credentials` grant directly from the v2.0 token endpoint:

```
TOKEN=$(curl -s -X POST \  "https://login.microsoftonline.com/$TENANT_ID/oauth2/v2.0/token" \  -d "grant_type=client_credentials" \  -d "client_id=$CLIENT_ID" \  -d "client_secret=$CLIENT_SECRET" \  -d "scope=api://flapi-saas/.default" \  | jq -r .access_token)
```

### 2\. Call the API

```
curl -s -H "Authorization: Bearer $TOKEN" \  "https://api.example.com/customers/?segment=enterprise&limit=10" \  | jq .
```

A valid token returns `200 OK` with tenant-scoped JSON:

```
{  "next": null,  "data": [    {      "customer_id": "c_8472",      "tenant_id": "t_acme",      "company_name": "Acme Pumps GmbH",      "segment": "enterprise",      "country": "DE"    }  ],  "total_count": 1}
```

### 3\. Inspect failures

Without a token:

```
curl -i https://api.example.com/customers/# HTTP/1.1 401 Unauthorized# WWW-Authenticate: Basic realm="flAPI"
```

With an expired or malformed token, the response is still `401 Unauthorized`. Run flAPI with `--log-level debug` to see the exact rejection cause from `OIDCAuthHandler::validateToken`:

*   `Failed to decode JWT` — not a JWT at all
*   `Token issuer mismatch` — `iss` claim disagrees with `auth.oidc.issuer-url` (often a tenant typo)
*   `Token audience validation failed` — token wasn't minted for this API
*   `Token has expired` — beyond `clock-skew-seconds`
*   `JWT signature verification failed` — wrong key, or JWKS cache is stale (retry after `jwks-cache-hours`)

## Claim Mapping in Practice

OIDC tokens vary wildly between providers in _where_ roles live. flAPI handles this with two knobs:

*   **`roles-claim`** — a flat claim name. The default is `roles`. Used when the token has `"roles": [...]` at the top level.
*   **`role-claim-path`** — a dot-separated path for nested claims. Used when the token has something like `"realm_access": { "roles": [...] }`.

When `role-claim-path` is set, `roles-claim` is ignored (see `OIDCAuthHandler::validateToken`). The dotted path supports arbitrary nesting; `getClaimArray` recurses on each `.` segment.

### Examples by provider

```
# Entra ID / Azure AD app roles — flatoidc:  provider-type: microsoft  roles-claim: roles# Keycloak realm roles — nestedoidc:  provider-type: keycloak  role-claim-path: realm_access.roles# Keycloak client roles for a specific client — nested deeperoidc:  provider-type: keycloak  role-claim-path: resource_access.my-api.roles# Auth0 with a custom namespaced claimoidc:  provider-type: auth0  role-claim-path: 'https://my-app.example.com/roles'
```

Inside SQL, every variant lands in the same place: `{{auth.roles}}`, a comma-joined string. The template never has to know which IdP issued the token.

## Troubleshooting Checklist

If a request is rejected and you don't know why, walk this list in order:

1.  **Decode the token offline** with `jq -R 'split(".")[1] | @base64d | fromjson'` and check `iss`, `aud`, `exp`, and the username claim.
2.  **Match `iss` against `auth.oidc.issuer-url`**, _substituting_ the placeholder. For Entra ID, `{tenant}` must equal the literal tenant GUID — not the domain (`contoso.onmicrosoft.com`) and not the friendly name.
3.  **Check `aud` against `allowed-audiences`** — Entra ID usually emits the App ID URI (`api://...`) when you're calling your own API; some flows emit the raw client GUID instead. List both if needed.
4.  **Confirm `exp` is in the future** with `date -u -d @$(jq -R 'split(".")[1] | @base64d | fromjson | .exp')`. If it's close, check NTP on the flAPI host.
5.  **Look up the `kid`** in the JWKS at `<issuer-url>/.well-known/openid-configuration` → `jwks_uri`. If the key isn't there, the IdP rotated it and flAPI's cache is stale — restart flAPI or wait for `jwks-cache-hours` to expire.
6.  **Inspect the role claim** — confirm `roles-claim` / `role-claim-path` matches the token's actual structure. A `null` from `getClaimArray` means the path didn't resolve.

## Operational Notes

*   **Key rotation**: JWKS rotation is handled automatically. When a token arrives with a `kid` not in the cache, flAPI re-fetches `jwks_uri` once and caches the new set.
*   **Multiple tenants**: With Entra ID's "multi-tenant" app registration, the literal `{tenant}` in the issuer template lets the same flAPI instance accept tokens from any Entra tenant that has consented to your app. Use `allowed-audiences` to keep the API surface bound to your client.
*   **Roles in the IdP**: Define Entra ID **app roles** on the API app registration, assign them to users / groups, and configure your token configuration to include them. They land in the `roles` claim and become `auth.roles` inside SQL.
*   **HTTPS**: The Authorization header carries a bearer token. Set `enforce-https.enabled: true` (already in the config above) and put flAPI behind a TLS terminator in production.

## See Also

*   **[Authentication reference](/docs/endpoints/authentication.md)** — every `auth.*` key, all three `auth.type` values (`basic`, `bearer`, `oidc`) plus the AWS Secrets Manager user source, with source-of-truth references.
*   **[CRUD API tutorial](/docs/examples/crud-api.md)** — same `request:` and `auth.username` patterns applied to writes.
*   **[AWS Secrets Manager credentials](/docs/examples/aws-secrets-rotation.md)** — sibling recipe for rotating _database_ credentials at the auth layer.

([🍪 Cookie Settings](#cookie-settings))

# Parquet File API Example

The simplest way to get started with flAPI: turn a Parquet file into a REST API in minutes.

## What You'll Build

A filterable customer API that:

*   ✅ Serves data from a local Parquet file
*   ✅ Supports query parameters
*   ✅ Validates input
*   ✅ Generates automatic documentation
*   ✅ Starts in milliseconds

## Prerequisites

*   flAPI installed ([Quickstart](/docs/getting-started/quickstart.md))
*   A Parquet file (we'll use sample customer data)

## How It Works

**Key advantage:** DuckDB reads Parquet files directly with no ETL, database, or import needed!

## Step-by-Step

### 1\. Project Structure

```
my-api/├── flapi.yaml├── data/│   └── customers.parquet└── sqls/    ├── customers.yaml    └── customers.sql
```

### 2\. Configuration

**`flapi.yaml`:**

```
project-name: customers-apiproject-description: Simple customer API from Parquettemplate:  path: './sqls'connections:  customers-data:    properties:      path: './data/customers.parquet'duckdb:  access_mode: READ_WRITE
```

### 3\. Endpoint Configuration

**`sqls/customers.yaml`:**

```
url-path: /customers/request:  - field-name: id    field-in: query    description: Customer ID    required: false    validators:      - type: int        min: 1    - field-name: segment    field-in: query    description: Market segment (AUTOMOBILE, BUILDING, FURNITURE)    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY]  - field-name: min_balance    field-in: query    description: Minimum account balance    required: false    validators:      - type: int        min: 0template-source: customers.sqlconnection:  - customers-data
```

### 4\. SQL Template

**`sqls/customers.sql`:**

```
SELECT  c_custkey as id,  c_name as name,  c_mktsegment as segment,  c_acctbal as balance,  c_address as address,  c_phone as phone,  c_comment as notesFROM '{{{conn.path}}}'WHERE 1=1{{#params.id}}  AND c_custkey = {{{params.id}}}{{/params.id}}{{#params.segment}}  AND c_mktsegment = '{{{params.segment}}}'{{/params.segment}}{{#params.min_balance}}  AND c_acctbal >= {{{params.min_balance}}}{{/params.min_balance}}ORDER BY c_acctbal DESCLIMIT 100
```

## Running the API

### Start the Server

```
$ ./flapi -c flapi.yaml✓ Loaded 1 endpoints✓ Server listening on :8080⚡ Ready in 1.2ms
```

### Test the Endpoints

**Get all customers:**

```
$ curl http://localhost:8080/customers/{  "data": [    {      "id": 1,      "name": "Customer#000000001",      "segment": "BUILDING",      "balance": 711.56,      "address": "IVhzIApeRb ot,c,E",      "phone": "25-989-741-2988",      "notes": "Regular requests..."    },    ...  ]}
```

**Filter by segment:**

```
$ curl "http://localhost:8080/customers/?segment=AUTOMOBILE"
```

**Filter by minimum balance:**

```
$ curl "http://localhost:8080/customers/?min_balance=5000"
```

**Get specific customer:**

```
$ curl "http://localhost:8080/customers/?id=12345"
```

**Combine filters:**

```
$ curl "http://localhost:8080/customers/?segment=AUTOMOBILE&min_balance=5000"
```

## Automatic Documentation

flAPI automatically generates OpenAPI (Swagger) documentation:

```
# Visit in your browserhttp://localhost:8080/docs
```

You'll see:

*   All endpoints listed
*   Parameter descriptions
*   Try-it-out functionality
*   Request/response examples

## Testing with CLI

Use the `flapii` CLI to test locally:

```
# Validate endpoint$ flapii endpoints validate /customers/✓ Configuration valid✓ SQL template found✓ All validators configured# Test template expansion$ flapii templates expand /customers/ \  --params '{"segment": "AUTOMOBILE", "min_balance": 5000}'Expanded SQL:SELECT  c_custkey as id,  c_name as name,  c_mktsegment as segment,  c_acctbal as balanceFROM './data/customers.parquet'WHERE 1=1  AND c_mktsegment = 'AUTOMOBILE'  AND c_acctbal >= 5000ORDER BY c_acctbal DESCLIMIT 100# Run query and see results$ flapii query run /customers/ \  --params '{"segment": "AUTOMOBILE"}' \  --limit 5
```

## Why This Works

### Performance

DuckDB reads Parquet files incredibly fast:

*   **Columnar storage**: Only reads needed columns
*   **Compression**: Efficient data storage
*   **Predicate pushdown**: Filters applied at read time
*   **Parallel processing**: Uses all CPU cores

### No Database Required

*   ✅ No PostgreSQL/MySQL to install
*   ✅ No server to maintain
*   ✅ Just files + flAPI
*   ✅ Perfect for prototypes and small-to-medium datasets

### Portability

```
# Everything in one directory$ tar -czf my-api.tar.gz my-api/$ scp my-api.tar.gz server:# On server$ tar -xzf my-api.tar.gz$ cd my-api && ./flapi -c flapi.yaml# API is running!
```

## Scaling Up

### Multiple Files

```
connections:  all-customers:    properties:      path: './data/customers/*.parquet'  # All files in directory
```

### Adding More Endpoints

```
# sqls/orders.yamlurl-path: /orders/connection:  - orders-data
```

### Adding Caching

If you get high traffic, add caching:

```
cache:  enabled: true  table: customers_cache  schedule: 60m
```

## Reuse field definitions with includes

Once you have more than one endpoint over the same dataset, you usually want the validators, rate limit, and connection settings in one place. flAPI supports YAML section includes with the syntax `{{include:section_name from path/to/file.yaml}}`, which inlines the named top-level section from the referenced file. The canonical example lives in [`examples/sqls/customers/customer-common.yaml`](https://github.com/datazoode/flapi/blob/main/examples/sqls/customers/customer-common.yaml) and consolidates the shared pieces:

sqls/common/customer-common.yaml

```
# Shared request validatorsrequest:  - field-name: id    field-in: query    description: Customer ID    required: false    validators:      - type: int        min: 1        max: 1000000  - field-name: segment    field-in: query    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY]  - field-name: email    field-in: query    required: false    validators:      - type: email# Shared connectionconnection:  - customers-data# Shared rate limitingrate-limit:  enabled: true  max: 100  interval: 60
```

The customer endpoint can now declare just what's specific to itself and pull the rest from the common file:

sqls/customers.yaml

```
url-path: /customers/{{include:request from common/customer-common.yaml}}{{include:rate-limit from common/customer-common.yaml}}{{include:connection from common/customer-common.yaml}}template-source: customers.sql
```

A second endpoint — say, a "VIP customers" view — picks up the exact same validators and rate limit without any copy-paste:

sqls/customers-vip.yaml

```
url-path: /customers/vip/{{include:request from common/customer-common.yaml}}{{include:rate-limit from common/customer-common.yaml}}{{include:connection from common/customer-common.yaml}}template-source: customers-vip.sql
```

When `segment` later needs a new allowed value, you update one file and every endpoint that includes it picks the change up on the next reload.

## Warm the cache with the heartbeat worker

flAPI ships with a background **heartbeat worker** that periodically pings endpoints you've opted in. Its job is to keep cached templates warm so the first real user request hits a fresh snapshot instead of paying the cold materialization cost. Enable it globally in `flapi.yaml`:

flapi.yaml

```
# Heartbeat configurationheartbeat:  enabled: true        # turn the worker on (off by default)  worker-interval: 10  # seconds between heartbeat passes
```

Then opt individual endpoints in. The `params:` map under the endpoint-level `heartbeat:` block is passed straight to the SQL template, so you can warm the most common filter combinations:

sqls/customers.yaml

```
url-path: /customers/# ... request, template-source, connection, cache ...heartbeat:  enabled: true  params:    segment: AUTOMOBILE   # pre-warm the most popular filter
```

## Next Steps

*   **[Quickstart Guide](/docs/getting-started/quickstart.md)**: Build your first API step-by-step
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Learn advanced Mustache templates
*   **[Endpoints Overview](/docs/endpoints/overview.md)**: Complete endpoint configuration
*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: When to use full vs. incremental cache modes (and how the heartbeat worker fits in)
*   **[BigQuery Example](/docs/examples/bigquery-caching.md)**: Scale to cloud warehouses with caching
*   **[SAP ERP Example](/docs/examples/sap-erp-api.md)**: Connect to enterprise systems
*   **[Parquet Connection Guide](/docs/guides/connections/parquet.md)**: Learn more about Parquet
*   **[Deployment](/docs/getting-started/deployment.md)**: Deploy to production

([🍪 Cookie Settings](#cookie-settings))

# SAP ERP Materials API

Build a fast, cost-effective materials inventory API from SAP ERP data. This example shows end-to-end configuration with caching, authentication, and MCP integration.

## Overview

**Use Case**: Provide real-time material inventory data to web applications, mobile apps, and AI agents without overloading SAP.

**Challenge**:

*   Direct SAP queries: 10-30 seconds
*   High RFC load on production SAP
*   Expensive ABAP custom development

**Solution**:

*   flAPI with 1-hour cache refresh
*   Millisecond API responses
*   Zero SAP load during API serving

## Project Structure

```
sap-materials-api/├── flapi.yaml                 # Main configuration├── sqls/│   ├── materials.yaml         # Endpoint config│   ├── materials.sql          # API query template│   └── materials_cache.sql    # Cache materialization└── .env                       # Environment variables (not committed)
```

## Step 1: Configuration

```
# flapi.yaml# ═══════════════════════════════════════════════════════════════# SAP Materials Inventory API# ═══════════════════════════════════════════════════════════════project-name: sap-materials-apiproject-description: Fast materials inventory API from SAP ERPtemplate:  path: './sqls'  environment-whitelist:    - '^SAP_.*'    - '^JWT_.*'duckdb:  threads: 4  max_memory: 4GB# DuckLake-backed cache for incremental refresh and snapshotsducklake:  enabled: true  alias: cache  metadata-path: ./data/cache.ducklake  data-path: ./data/cacheconnections:  sap-prod:    init: |      INSTALL 'erpl' FROM 'http://get.erpl.io';      LOAD 'erpl';      CREATE OR REPLACE PERSISTENT SECRET sap_prod (        TYPE sap_rfc,        ASHOST '{{env.SAP_HOST}}',        SYSNR  '{{env.SAP_SYSNR}}',        CLIENT '{{env.SAP_CLIENT}}',        USER   '{{env.SAP_USER}}',        PASSWD '{{env.SAP_PASSWORD}}',        LANG   'EN'      );enforce-https:  enabled: true
```

## Step 2: Cache Template

```
-- sqls/materials_cache.sql-- ═══════════════════════════════════════════════════════════════-- Materialize SAP material master and inventory data.-- flAPI executes this SELECT on the configured schedule and writes the-- result into {{cache.catalog}}.{{cache.schema}}.{{cache.table}}.-- ═══════════════════════════════════════════════════════════════WITH  mara AS (SELECT * FROM sap_read_table('MARA')),  makt AS (SELECT * FROM sap_read_table('MAKT') WHERE SPRAS = 'E'),  marc AS (SELECT * FROM sap_read_table('MARC')),  mard AS (SELECT * FROM sap_read_table('MARD')),  mbew AS (SELECT * FROM sap_read_table('MBEW'))SELECT  -- Material master (MARA)  m.MATNR as material_number,  m.MTART as material_type,  m.MATKL as material_group,  m.MEINS as base_unit_of_measure,  m.BRGEW as gross_weight,  m.NTGEW as net_weight,  m.GEWEI as weight_unit,  -- Material description (MAKT)  t.MAKTX as material_description,  t.MAKTG as material_description_short,  -- Plant data (MARC)  p.WERKS as plant,  p.DISPO as mrp_controller,  p.DISMM as mrp_type,  p.PLIFZ as planned_delivery_time,  -- Storage location data (MARD)  s.LGORT as storage_location,  s.LABST as unrestricted_stock,  s.INSME as quality_inspection_stock,  s.SPEME as blocked_stock,  s.UMLME as stock_in_transfer,  -- Valuation (MBEW)  v.STPRS as standard_price,  v.PEINH as price_unit,  v.BKLAS as valuation_class,  v.SALK3 as total_stock_value,  -- Calculated fields  (s.LABST + s.INSME) as available_stock,  CASE    WHEN s.LABST = 0 THEN 'OUT_OF_STOCK'    WHEN s.LABST < 10 THEN 'LOW_STOCK'    ELSE 'IN_STOCK'  END as stock_status,  CURRENT_TIMESTAMP as cache_updated_atFROM mara mLEFT JOIN makt t ON m.MATNR = t.MATNRLEFT JOIN marc p ON m.MATNR = p.MATNRLEFT JOIN mard s ON m.MATNR = s.MATNR AND p.WERKS = s.WERKSLEFT JOIN mbew v ON m.MATNR = v.MATNR AND p.WERKS = v.BWKEYWHERE m.LVORM IS NULL  AND (m.LOEKZ IS NULL OR m.LOEKZ = '')  AND (s.LABST > 0 OR p.MMSTA = 'A')ORDER BY m.MATNR, p.WERKS, s.LGORT
```

## Step 3: Endpoint Configuration

```
# sqls/materials.yaml# ═══════════════════════════════════════════════════════════════# Materials Inventory API Endpoint# GET /materials/?plant=1000&material_type=FERT# ═══════════════════════════════════════════════════════════════url-path: /materials/method: GETdescription: |  Get material inventory data from SAP ERP with millisecond response times.  Data is refreshed hourly from SAP production system.# ───────────────────────────────────────────────────────────────# Authentication (per-endpoint, single auth type)# ───────────────────────────────────────────────────────────────auth:  enabled: true  type: bearer  jwt-secret: '{{env.JWT_SECRET}}'  jwt-issuer: my-auth-server# ───────────────────────────────────────────────────────────────# Rate Limiting (SAP RFC calls are expensive — protect production)# ───────────────────────────────────────────────────────────────rate-limit:  enabled: true  max: 30                    # 30 requests  interval: 60               # per 60-second window per client# ───────────────────────────────────────────────────────────────# Caching Configuration (Essential for SAP)# ───────────────────────────────────────────────────────────────cache:  enabled: true  table: sap_materials_inventory  schema: analytics  schedule: 60m              # Refresh every hour  template-file: materials_cache.sql# ───────────────────────────────────────────────────────────────# Request Parameters# ───────────────────────────────────────────────────────────────request:  - field-name: plant    field-in: query    description: Plant code (e.g., 1000, 2000)    required: false    validators:      - type: string        regex: '^\d{4}$'  - field-name: material_type    field-in: query    description: Material type (FERT=Finished goods, HALB=Semi-finished, ROH=Raw materials)    required: false    validators:      - type: enum        allowedValues: [FERT, HALB, ROH, HIBE]  - field-name: material_group    field-in: query    description: Material group code    required: false    validators:      - type: string        max: 50  - field-name: stock_status    field-in: query    description: Stock availability status    required: false    validators:      - type: enum        allowedValues: [IN_STOCK, LOW_STOCK, OUT_OF_STOCK]  - field-name: search    field-in: query    description: Search in material number or description    required: false    validators:      - type: string        min: 3        max: 100# ───────────────────────────────────────────────────────────────# SQL Template# ───────────────────────────────────────────────────────────────template-source: materials.sqlconnection:  - sap-prod# Built-in pagination provides 'limit' and 'offset' parameters.with-pagination: true# ───────────────────────────────────────────────────────────────# MCP Tool Configuration (for AI agents)# ───────────────────────────────────────────────────────────────mcp-tool:  name: get_sap_materials  description: |    Query SAP ERP materials inventory data. Returns material master information,    stock levels, pricing, and availability status.    Use this tool when you need to:    - Check material availability and stock levels    - Get material descriptions and specifications    - Find materials by plant or material type    - Analyze inventory across locations  result-mime-type: application/json
```

## Step 4: API Query Template

```
-- sqls/materials.sql-- ═══════════════════════════════════════════════════════════════-- Fast queries against materialized cache-- ═══════════════════════════════════════════════════════════════SELECT   material_number,  material_description,  material_type,  material_group,  plant,  storage_location,  unrestricted_stock,  quality_inspection_stock,  blocked_stock,  available_stock,  stock_status,  base_unit_of_measure,  standard_price,  price_unit,  total_stock_value,  mrp_controller,  planned_delivery_time,  cache_updated_atFROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE 1=1{{#params.plant}}  AND plant = '{{{params.plant}}}'{{/params.plant}}{{#params.material_type}}  AND material_type = '{{{params.material_type}}}'{{/params.material_type}}{{#params.material_group}}  AND material_group = '{{{params.material_group}}}'{{/params.material_group}}{{#params.stock_status}}  AND stock_status = '{{{params.stock_status}}}'{{/params.stock_status}}{{#params.search}}  AND (    material_number LIKE '%{{{params.search}}}%'    OR LOWER(material_description) LIKE LOWER('%{{{params.search}}}%')  ){{/params.search}}ORDER BY  plant,  material_number
```

When `with-pagination: true` is set on the endpoint, flAPI automatically applies `LIMIT` / `OFFSET` from the `?limit=…&offset=…` query parameters and returns pagination metadata, so the template does not need to handle it.

## Step 5: Environment Configuration

```
# .env (NEVER commit to git)# ═══════════════════════════════════════════════════════════════# SAP ERP Connection# ═══════════════════════════════════════════════════════════════export SAP_HOST="sap-prod.company.com"export SAP_SYSNR="00"export SAP_CLIENT="100"export SAP_USER="RFC_API_USER"export SAP_PASSWORD="secure-sap-password"# ═══════════════════════════════════════════════════════════════# Security# ═══════════════════════════════════════════════════════════════export JWT_SECRET="your-super-secret-jwt-key-min-32-chars"# ═══════════════════════════════════════════════════════════════# Load environment# ═══════════════════════════════════════════════════════════════source .env
```

## Step 6: Run flAPI

```
# Start the server with the flAPI binary, pointing at your config file./flapi -c flapi.yaml# Output:# [INFO] Loading configuration...# [INFO] Connecting to SAP ERP...# [INFO] Initializing cache: sap_materials_inventory# [INFO] Cache refresh scheduled: every 60 minutes# [INFO] Server running on http://0.0.0.0:8080
```

## Step 7: Test the API

### Basic Query

```
curl -H "Authorization: Bearer $TOKEN" \  "http://localhost:8080/materials/?limit=5"
```

**Response:**

```
{  "data": [    {      "material_number": "000000000010000001",      "material_description": "Centrifugal Pump Model X100",      "material_type": "FERT",      "material_group": "PUMPS",      "plant": "1000",      "storage_location": "0001",      "unrestricted_stock": 45.0,      "quality_inspection_stock": 0.0,      "blocked_stock": 2.0,      "available_stock": 45.0,      "stock_status": "IN_STOCK",      "base_unit_of_measure": "EA",      "standard_price": 1250.00,      "price_unit": 1,      "total_stock_value": 56250.00,      "mrp_controller": "001",      "planned_delivery_time": 14,      "cache_updated_at": "2024-01-15T10:00:00Z"    }  ],  "metadata": {    "total": 2847,    "limit": 5,    "offset": 0  }}
```

### Filter by Plant

```
curl -H "Authorization: Bearer $TOKEN" \  "http://localhost:8080/materials/?plant=1000&material_type=FERT"
```

### Search Materials

```
curl -H "Authorization: Bearer $TOKEN" \  "http://localhost:8080/materials/?search=pump&limit=10"
```

### Check Low Stock

```
curl -H "Authorization: Bearer $TOKEN" \  "http://localhost:8080/materials/?stock_status=LOW_STOCK"
```

## Performance Comparison

| Method | Query Time | Concurrent Users | SAP Load |
| --- | --- | --- | --- |
| **Direct SAP RFC** | 10-30 seconds | 5-10 | High |
| **flAPI Cached** | 1-5ms | 10,000+ | Minimal (hourly refresh) |

**Cost Savings:**

*   SAP RFC load reduced by 99%
*   Response time improved by 10,000x
*   Can serve millions of API calls with single hourly SAP query

## AI Agent Integration (MCP)

```
// Claude/GPT can now query SAP inventoryconst response = await mcp.call("get_sap_materials", {  plant: "1000",  stock_status: "LOW_STOCK"});console.log(`Found ${response.data.length} low stock materials`);
```

### Expose the materials schema as an MCP resource

The `mcp-tool` block above lets an agent **call** the materials endpoint. Agents also benefit from a static **resource** that describes the result shape up front, so they can plan queries without trial-and-error. Drop a second YAML file alongside `materials.yaml` to publish one:

sqls/materials-mcp-resource.yaml

```
mcp-resource:  name: materials_schema  description: |    JSON schema definition for the SAP materials inventory endpoint —    field names, SQL types, and human-readable descriptions. Use this    resource before calling the get_sap_materials tool so you know what    columns and filters are available.  mime-type: application/jsontemplate-source: materials-schema.sqlconnection:  - sap-prod# Resources are read by tooling, not humans — light rate limit is plenty.rate-limit:  enabled: true  max: 10  interval: 60
```

The companion SQL template returns a JSON document describing the columns the materials endpoint serves:

sqls/materials-schema.sql

```
SELECT json_object(  'endpoint', '/materials/',  'description', 'SAP material master + inventory + valuation',  'columns', json_array(    json_object('name', 'material_number', 'type', 'VARCHAR', 'description', 'SAP material number (MARA-MATNR)'),    json_object('name', 'material_description', 'type', 'VARCHAR', 'description', 'Long description (MAKT-MAKTX, English)'),    json_object('name', 'material_type', 'type', 'VARCHAR', 'description', 'FERT=finished, HALB=semi-finished, ROH=raw, HIBE=operating supplies'),    json_object('name', 'material_group', 'type', 'VARCHAR', 'description', 'Material group (MARA-MATKL)'),    json_object('name', 'plant', 'type', 'VARCHAR', 'description', '4-digit plant code (MARC-WERKS)'),    json_object('name', 'storage_location', 'type', 'VARCHAR', 'description', 'Storage location within plant (MARD-LGORT)'),    json_object('name', 'unrestricted_stock', 'type', 'DOUBLE', 'description', 'Unrestricted-use stock quantity (MARD-LABST)'),    json_object('name', 'available_stock', 'type', 'DOUBLE', 'description', 'unrestricted + quality-inspection stock'),    json_object('name', 'stock_status', 'type', 'VARCHAR', 'description', 'IN_STOCK, LOW_STOCK, or OUT_OF_STOCK'),    json_object('name', 'standard_price', 'type', 'DOUBLE', 'description', 'Standard price per price_unit (MBEW-STPRS)'),    json_object('name', 'cache_updated_at', 'type', 'TIMESTAMP', 'description', 'When the cached snapshot was materialized')  ),  'filters', json_array('plant', 'material_type', 'material_group', 'stock_status', 'search')) as schema_definition;
```

An agent connected over MCP now sees both a `get_sap_materials` tool and a `materials_schema` resource. Most coding agents fetch resources up front, so calls to the tool arrive already correctly typed.

## Next Steps

*   **[SAP ERP Connection Guide](/docs/guides/connections/sap-erp.md)**: Detailed SAP ERP setup
*   **[SAP BW Connection](/docs/guides/connections/sap-bw.md)**: Connect to SAP BW/4HANA
*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: Understand optimization techniques
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Configure caching for SAP
*   **[Authentication](/docs/endpoints/authentication.md)**: Secure production deployment
*   **[MCP Protocol](/docs/ai-integration/mcp-protocol.md)**: How tools, resources, and prompts fit together
*   **[MCP Resources recipe](/docs/examples/mcp-resources.md)**: More patterns for exposing schemas and reference data to agents
*   **[BigQuery Example](/docs/examples/bigquery-caching.md)**: Cloud warehouse alternative
*   **[Deployment](/docs/getting-started/deployment.md)**: Deploy to production

([🍪 Cookie Settings](#cookie-settings))

# RAG Knowledge Base API

**Extension Credit:**

This example uses the **[VSS (Vector Similarity Search)](https://duckdb.org/docs/stable/core_extensions/vss.html)** core extension and **[faiss extension](https://duckdb.org/community_extensions/extensions/faiss.html)** by the DuckDB community. Thanks to the contributors who made vector search accessible in DuckDB!

Build a production-ready RAG (Retrieval Augmented Generation) system that gives AI agents instant access to your company knowledge. This example shows how to create a searchable documentation API that powers AI-driven customer support.

## Use Case

**Scenario:** A SaaS company wants an AI chatbot that can answer customer questions using their documentation, but:

*   LLMs hallucinate without context
*   Documentation changes frequently
*   Need sub-second retrieval for good UX
*   Want to expose via REST API + MCP for AI agents

**Solution:** Build a vector search API with flAPI that:

*   ✅ Stores documentation as embeddings
*   ✅ Provides semantic search (finds by meaning)
*   ✅ Returns relevant context for LLMs
*   ✅ Exposes via REST + MCP tools
*   ✅ Updates easily when docs change

## Architecture

**Flow:**

1.  Documentation → Generate embeddings → Store in DuckDB
2.  User asks question → Embed question → Search vectors
3.  Return relevant docs → LLM generates answer

## Step 1: Generate Embeddings

### Install Dependencies

```
pip install openai duckdb tiktoken
```

### Embedding Script

generate\_embeddings.py

```
"""Generate embeddings for documentation and store in DuckDB"""import osimport jsonimport duckdbimport openaifrom pathlib import Pathfrom typing import List, Dict# Configure OpenAIopenai.api_key = os.getenv("OPENAI_API_KEY")# Connect to DuckDBconn = duckdb.connect('knowledge_base.duckdb')# Create table with vector supportconn.execute("""    INSTALL vss;    LOAD vss;        CREATE TABLE IF NOT EXISTS documents (        id VARCHAR PRIMARY KEY,        title VARCHAR,        content TEXT,        section VARCHAR,        url VARCHAR,        embedding FLOAT[1536],  -- OpenAI ada-002 dimension        metadata JSON,        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,        updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP    );        -- Create HNSW index for fast similarity search    CREATE INDEX IF NOT EXISTS doc_embedding_idx     ON documents USING HNSW (embedding);""")def chunk_text(text: str, max_tokens: int = 500) -> List[str]:    """    Split text into chunks for embedding    OpenAI has 8191 token limit, we use 500 for good granularity    """    # Simple sentence-based chunking    sentences = text.split('. ')    chunks = []    current_chunk = []    current_length = 0        for sentence in sentences:        sentence_length = len(sentence.split())        if current_length + sentence_length > max_tokens:            chunks.append('. '.join(current_chunk) + '.')            current_chunk = [sentence]            current_length = sentence_length        else:            current_chunk.append(sentence)            current_length += sentence_length        if current_chunk:        chunks.append('. '.join(current_chunk))        return chunksdef generate_embedding(text: str) -> List[float]:    """Generate embedding using OpenAI"""    response = openai.Embedding.create(        input=text,        model="text-embedding-3-small"  # Cheaper than ada-002    )    return response['data'][0]['embedding']def process_documentation(docs_dir: Path):    """Process all markdown files in docs directory"""        doc_count = 0        for md_file in docs_dir.glob("**/*.md"):        print(f"Processing: {md_file}")                content = md_file.read_text()                # Extract title (first # heading)        title = "Untitled"        for line in content.split('\n'):            if line.startswith('# '):                title = line[2:].strip()                break                # Determine section from path        section = md_file.parent.name                # Generate URL        url = f"/docs/{md_file.relative_to(docs_dir).with_suffix('')}"                # Chunk content        chunks = chunk_text(content)                for i, chunk in enumerate(chunks):            if len(chunk.strip()) < 50:  # Skip tiny chunks                continue                        doc_id = f"{md_file.stem}_chunk_{i}"                        # Generate embedding            embedding = generate_embedding(chunk)                        # Insert into database            conn.execute("""                INSERT OR REPLACE INTO documents                 (id, title, content, section, url, embedding, metadata)                VALUES (?, ?, ?, ?, ?, ?, ?)            """, [                doc_id,                title,                chunk,                section,                url,                embedding,                json.dumps({                    'file': str(md_file),                    'chunk_index': i,                    'total_chunks': len(chunks)                })            ])                        doc_count += 1            print(f"  - Chunk {i+1}/{len(chunks)}: {len(chunk)} chars")        print(f"\n✅ Processed {doc_count} document chunks")    conn.close()if __name__ == "__main__":    # Process your documentation    docs_dir = Path("./docs")        if not docs_dir.exists():        print(f"❌ Documentation directory not found: {docs_dir}")        exit(1)        process_documentation(docs_dir)
```

### Run Embedding Generation

```
# Set your OpenAI API keyexport OPENAI_API_KEY="sk-..."# Generate embeddingspython generate_embeddings.py# Output:# Processing: docs/getting-started/quickstart.md#   - Chunk 1/3: 512 chars#   - Chunk 2/3: 487 chars#   - Chunk 3/3: 301 chars# ...# ✅ Processed 127 document chunks
```

**Cost estimate:**

*   100 docs × 3 chunks × 500 tokens = 150K tokens
*   Cost: $0.003 (text-embedding-3-small)

## Step 2: Configure flAPI

### Main Configuration

flapi.yaml

```
project-name: rag-knowledge-baseproject-description: Vector search over documentation embeddingstemplate:  path: './sqls'  environment-whitelist:    - '^OPENAI_.*'connections:  vector-db:    init: |      -- Load VSS extension and attach the embeddings database      INSTALL vss;      LOAD vss;      ATTACH IF NOT EXISTS './knowledge_base.duckdb' AS kb;    properties:      database_path: './knowledge_base.duckdb'duckdb:  access_mode: READ_WRITE
```

## Step 3: Create Search Endpoints

### Endpoint 1: Semantic Search

sqls/semantic\_search.sql

```
-- Search documentation by semantic similarityWITH query_vector AS (  -- In production, pass pre-computed embedding  SELECT CAST({{{params.embedding}}} AS FLOAT[1536]) as query_emb)SELECT  d.id,  d.title,  d.content,  d.section,  d.url,  d.metadata,  -- Calculate cosine similarity  array_cosine_similarity(d.embedding, q.query_emb) as similarityFROM kb.documents d, query_vector qWHERE 1=1  -- Fixed similarity threshold (0-1, higher = more similar)  AND array_cosine_similarity(d.embedding, q.query_emb) >= 0.7  -- Optional section filter  {{#params.section}}  AND d.section = '{{{params.section}}}'  {{/params.section}}  -- Optional keyword filter (hybrid search)  {{#params.keywords}}  AND LOWER(d.content) LIKE LOWER('%{{{params.keywords}}}%')  {{/params.keywords}}ORDER BY similarity DESCLIMIT {{#params.limit}}{{{params.limit}}}{{/params.limit}}{{^params.limit}}5{{/params.limit}}
```

sqls/semantic\_search.yaml

```
url-path: /search/semantic/method: POSTdescription: Semantic search over documentation using vector embeddingstemplate-source: semantic_search.sqlconnection:  - vector-db# Vector cosine-similarity passes over every row in the table — keep callers# from accidentally turning the API into a runaway batch job.rate-limit:  enabled: true  max: 60                # 60 requests  interval: 60           # per minute, per client# Also expose this as an MCP tool for AI agentsmcp-tool:  name: search_documentation  description: |    Search the knowledge base semantically to find relevant documentation    for answering user questions. Uses vector embeddings to find documentation    by meaning, not just keywords. Use this to gather context before answering.  result-mime-type: application/jsonrequest:  - field-name: embedding    field-in: body    description: Query embedding as JSON array [1536 dimensions]    required: true    validators:      - type: string        # Strings holding a JSON array contain commas and brackets; the        # default SQL-injection guard is too strict for this payload.        preventSqlInjection: false  - field-name: section    field-in: query    description: Filter by documentation section    required: false    validators:      - type: enum        allowedValues: [getting-started, guides, api, examples, advanced]  - field-name: keywords    field-in: query    description: Additional keyword filter for hybrid search    required: false    validators:      - type: string        max: 200  - field-name: limit    field-in: query    description: Maximum number of results (default 5)    required: false    validators:      - type: int        min: 1        max: 20
```

### Endpoint 2: Browse by Section

sqls/browse\_sections.sql

```
-- Get all available sections with document countsSELECT  section,  COUNT(DISTINCT id) as doc_count,  COUNT(DISTINCT title) as unique_titles,  MIN(created_at) as oldest_doc,  MAX(updated_at) as newest_docFROM kb.documentsWHERE section IS NOT NULLGROUP BY sectionORDER BY doc_count DESC
```

sqls/browse\_sections.yaml

```
url-path: /search/sections/template-source: browse_sections.sqlconnection:  - vector-dbdescription: List all documentation sections with counts
```

### Endpoint 3: Get Document by ID

sqls/get\_document.sql

```
SELECT  id,  title,  content,  section,  url,  metadata,  created_at,  updated_atFROM kb.documentsWHERE id = '{{{params.doc_id}}}'LIMIT 1
```

sqls/get\_document.yaml

```
url-path: /documents/:doc_id/method: GETtemplate-source: get_document.sqlconnection:  - vector-dbrequest:  - field-name: doc_id    field-in: path    description: Document ID    required: true    validators:      - type: string        max: 200
```

### Endpoint 4: Related Documents

sqls/related\_docs.sql

```
-- Find documents similar to a given documentWITH source_doc AS (  SELECT embedding  FROM kb.documents  WHERE id = '{{{params.doc_id}}}')SELECT  d.id,  d.title,  d.content,  d.section,  d.url,  array_cosine_similarity(d.embedding, s.embedding) as similarityFROM kb.documents d, source_doc sWHERE d.id != '{{{params.doc_id}}}'  AND array_cosine_similarity(d.embedding, s.embedding) >= 0.8ORDER BY similarity DESCLIMIT {{#params.limit}}{{{params.limit}}}{{/params.limit}}{{^params.limit}}5{{/params.limit}}
```

sqls/related\_docs.yaml

```
url-path: /documents/:doc_id/related/method: GETtemplate-source: related_docs.sqlconnection:  - vector-dbrequest:  - field-name: doc_id    field-in: path    description: Source document ID    required: true    validators:      - type: string        max: 200  - field-name: limit    field-in: query    description: Max results (default 5)    required: false    validators:      - type: int        min: 1        max: 20
```

## Step 4: Client Implementation

### Python Client with OpenAI

rag\_client.py

```
"""RAG Client - Search documentation and generate answers"""import requestsimport openaiimport osopenai.api_key = os.getenv("OPENAI_API_KEY")FLAPI_URL = "http://localhost:8080"def search_documentation(query: str, limit: int = 5) -> list:    """Search documentation semantically"""        # 1. Generate embedding for query    embedding_response = openai.Embedding.create(        input=query,        model="text-embedding-3-small"    )    query_embedding = embedding_response['data'][0]['embedding']        # 2. Search via flAPI    response = requests.post(        f"{FLAPI_URL}/search/semantic/",        json={            'embedding': query_embedding,            'limit': limit,            'min_similarity': 0.7        }    )        if response.status_code != 200:        raise Exception(f"Search failed: {response.text}")        return response.json()['data']def generate_answer(question: str, context_docs: list) -> str:    """Generate answer using LLM with context"""        # Build context from retrieved documents    context = "\n\n---\n\n".join([        f"Document: {doc['title']}\n"        f"Section: {doc['section']}\n"        f"URL: {doc['url']}\n"        f"Content: {doc['content']}\n"        f"Relevance: {doc['similarity']:.2f}"        for doc in context_docs    ])        # Generate answer with GPT-4    response = openai.ChatCompletion.create(        model="gpt-4-turbo-preview",        messages=[            {                "role": "system",                "content": (                    "You are a helpful documentation assistant. "                    "Answer questions using ONLY the provided context. "                    "If the answer isn't in the context, say so. "                    "Always cite the document URL in your answer."                )            },            {                "role": "user",                "content": f"Question: {question}\n\nContext:\n{context}"            }        ],        temperature=0.3  # Lower = more factual    )        return response.choices[0].message.contentdef answer_question(question: str) -> dict:    """Complete RAG flow: search + generate"""        print(f"🔍 Searching for: {question}")        # 1. Search documentation    docs = search_documentation(question, limit=5)        if not docs:        return {            'answer': "I couldn't find relevant information to answer your question.",            'sources': []        }        print(f"📚 Found {len(docs)} relevant documents")        # 2. Generate answer    print("🤖 Generating answer...")    answer = generate_answer(question, docs)        return {        'answer': answer,        'sources': [            {                'title': doc['title'],                'url': doc['url'],                'similarity': doc['similarity']            }            for doc in docs        ]    }if __name__ == "__main__":    # Example usage    questions = [        "How do I connect to BigQuery?",        "What is caching and how does it work?",        "How do I authenticate API requests?"    ]        for question in questions:        result = answer_question(question)        print(f"\n{'='*60}")        print(f"Q: {question}")        print(f"{'='*60}")        print(f"A: {result['answer']}\n")        print("Sources:")        for source in result['sources']:            print(f"  - {source['title']} ({source['similarity']:.2f}): {source['url']}")        print()
```

### Test the Client

```
python rag_client.py# Output:# ============================================================# Q: How do I connect to BigQuery?# ============================================================# 🔍 Searching for: How do I connect to BigQuery?# 📚 Found 5 relevant documents# 🤖 Generating answer...# A: To connect to BigQuery, you need to install the BigQuery extension# and configure your connection with project credentials. Here's how:## 1. Install the extension: `INSTALL 'bigquery'; LOAD 'bigquery';`# 2. Configure connection with your project_id and credentials# 3. Use bigquery_scan() to query tables## See: /docs/guides/connections/bigquery for complete setup instructions.## Sources:#   - BigQuery Connection (0.89): /docs/guides/connections/bigquery#   - Getting Started (0.82): /docs/getting-started/quickstart#   - Connections Overview (0.78): /docs/guides/connections/overview
```

## Step 5: MCP Integration for AI Agents

### MCP Server Setup

mcp\_server.py

```
"""MCP Server - Expose RAG as tools for Claude/GPT"""from mcp import MCPServerimport openaiimport requestsimport osmcp = MCPServer(name="knowledge-base")openai.api_key = os.getenv("OPENAI_API_KEY")FLAPI_URL = "http://localhost:8080"@mcp.tool("search_documentation")async def search_docs(    query: str,    section: str = None,    limit: int = 5) -> list:    """    Search the knowledge base semantically.        Args:        query: The search query or question        section: Optional section filter        limit: Max results (default 5)        Returns:        List of relevant documents with content and URLs    """        # Generate embedding    embedding_response = openai.Embedding.create(        input=query,        model="text-embedding-3-small"    )    embedding = embedding_response['data'][0]['embedding']        # Search via flAPI    payload = {        'embedding': embedding,        'limit': limit,        'min_similarity': 0.7    }        if section:        payload['section'] = section        response = requests.post(        f"{FLAPI_URL}/search/semantic/",        json=payload    )        if response.status_code != 200:        return []        return response.json()['data']@mcp.tool("list_sections")async def list_sections() -> list:    """    List all available documentation sections.        Returns:        List of sections with document counts    """    response = requests.get(f"{FLAPI_URL}/search/sections/")        if response.status_code != 200:        return []        return response.json()['data']@mcp.tool("get_related_docs")async def get_related(doc_id: str, limit: int = 5) -> list:    """    Find documents related to a given document.        Args:        doc_id: Source document ID        limit: Max results (default 5)        Returns:        List of related documents    """    response = requests.get(        f"{FLAPI_URL}/documents/{doc_id}/related/",        params={'limit': limit}    )        if response.status_code != 200:        return []        return response.json()['data']if __name__ == "__main__":    # Run MCP server    mcp.run()
```

### Claude Desktop Configuration

~/Library/Application Support/Claude/claude\_desktop\_config.json

```
{  "mcpServers": {    "knowledge-base": {      "command": "python",      "args": ["/path/to/mcp_server.py"],      "env": {        "OPENAI_API_KEY": "sk-..."      }    }  }}
```

### Using with Claude

```
You: "What's the best way to cache BigQuery data in flAPI?"Claude: [Uses search_documentation tool]Let me search our documentation for BigQuery caching strategies...[Tool: search_documentation]query: "BigQuery caching strategies"section: "guides"Based on the documentation, here's the best approach for caching BigQuery data:1. **Use cache-template** with scheduled refreshes2. **Set appropriate intervals** (every 4-6 hours for most use cases)3. **Implement incremental refresh** for large datasets4. **Use smallest warehouse** for cache refreshesExample configuration:...Sources:- BigQuery Connection Guide: /docs/guides/connections/bigquery- Caching Strategy: /docs/concepts/caching-strategy
```

### Ship an MCP prompt that orchestrates the workflow

Tools let an agent retrieve data, but **MCP prompts** let you ship the _workflow_ that uses them. Drop a separate YAML file that defines a prompt the host application (Claude Desktop, an internal copilot, etc.) can list for end-users. The agent receives the rendered template along with the arguments it was invoked with:

sqls/documentation\_search\_workflow.yaml

```
mcp-prompt:  name: documentation_search_workflow  description: |    Guide an agent through a multi-step documentation lookup: search,    optionally pull in related docs, and synthesize a cited answer.  template: |    You are answering a question against the project documentation.    ## User question    {{query}}    ## Workflow    1. Call the `search_documentation` MCP tool with the question above.       {{#max_results}}       Request up to {{max_results}} results.       {{/max_results}}       {{^max_results}}       Request the default top 5 results.       {{/max_results}}    2. Inspect the returned documents. If similarity scores are all below       0.75, broaden the query or fall back to a keyword filter.    {{#include_related}}    3. For the highest-scoring document, call the related-documents       endpoint at `/documents/:doc_id/related/` to pull adjacent context.    {{/include_related}}    4. Draft an answer using ONLY the retrieved content. Every claim must       cite the source document's URL.    Return the final answer in markdown with a "Sources" section at the end.  # Arguments the host application may pass when invoking the prompt  arguments:    - query    - max_results    - include_related
```

The host UI will render `query` as a required text input and the other two as optional toggles. The same Mustache rules used in SQL templates apply here, so `{{#max_results}}…{{/max_results}}` only renders when the caller supplied the argument and `{{^max_results}}…{{/max_results}}` renders the fallback otherwise.

## Step 6: Web UI (Optional)

### Simple React Search Interface

SearchInterface.jsx

```
import React, { useState } from 'react';function SearchInterface() {  const [query, setQuery] = useState('');  const [results, setResults] = useState([]);  const [answer, setAnswer] = useState('');  const [loading, setLoading] = useState(false);  async function handleSearch() {    setLoading(true);        try {      // Call your RAG client API      const response = await fetch('/api/rag/answer', {        method: 'POST',        headers: { 'Content-Type': 'application/json' },        body: JSON.stringify({ question: query })      });            const data = await response.json();      setAnswer(data.answer);      setResults(data.sources);    } catch (error) {      console.error('Search failed:', error);    } finally {      setLoading(false);    }  }  return (    <div className="search-container">      <h1>📚 Search Documentation</h1>            <div className="search-box">        <input          type="text"          value={query}          onChange={(e) => setQuery(e.target.value)}          placeholder="Ask a question..."          onKeyPress={(e) => e.key === 'Enter' && handleSearch()}        />        <button onClick={handleSearch} disabled={loading}>          {loading ? 'Searching...' : 'Search'}        </button>      </div>      {answer && (        <div className="answer">          <h2>Answer</h2>          <p>{answer}</p>                    <h3>Sources</h3>          <ul>            {results.map((source, i) => (              <li key={i}>                <a href={source.url} target="_blank">                  {source.title}                </a>                <span className="similarity">                  {(source.similarity * 100).toFixed(0)}% relevant                </span>              </li>            ))}          </ul>        </div>      )}    </div>  );}export default SearchInterface;
```

## Performance Metrics

### Real-World Benchmarks

**Setup:**

*   127 documentation chunks (50KB total)
*   OpenAI text-embedding-3-small (1536 dimensions)
*   DuckDB with HNSW index
*   M1 MacBook Pro

**Results:**

| Operation | Time | Notes |
| --- | --- | --- |
| **Initial embedding generation** | 45s | One-time (127 docs × 350ms each) |
| **Vector search (cold)** | 12ms | First query after server start |
| **Vector search (warm)** | 2-5ms | Subsequent queries |
| **Full RAG flow** | 1.8s | Search (5ms) + GPT-4 (1.8s) |

**Comparison with alternatives:**

| Approach | Query Time | Setup Complexity |
| --- | --- | --- |
| **Pinecone** | 50-100ms | Medium (external service) |
| **Weaviate** | 20-50ms | High (separate server) |
| **flAPI + DuckDB** | **2-5ms** | Low (single binary) |

## Cost Analysis

### Monthly Operating Costs

**Assumptions:**

*   100 documentation pages
*   10,000 searches/month
*   1,000 RAG answers/month (with GPT-4)

**Costs:**

| Component | Usage | Cost |
| --- | --- | --- |
| **Embeddings (initial)** | 100 docs × 3 chunks × 500 tokens | $0.005 |
| **Embeddings (updates)** | 10 docs/month × 3 chunks | $0.0005/mo |
| **Search queries** | 10,000 × free (local DuckDB) | $0  |
| **LLM (GPT-4)** | 1,000 answers × $0.03 | $30/mo |
| **flAPI hosting** | Fly.io / Railway | $5-10/mo |
| **Total** |     | **~$35-40/mo** |

**vs. Alternatives:**

*   **Pinecone**: $70/mo (starter) + embeddings + LLM = $100+/mo
*   **OpenAI Assistants**: $0.03/query × 10K = $300/mo
*   **Custom build**: Dev time (20-40 hours) + infrastructure

## Updating Documentation

### Incremental Updates

update\_docs.py

```
"""Update embeddings when documentation changes"""import duckdbimport openaiconn = duckdb.connect('knowledge_base.duckdb')def update_document(doc_id: str, new_content: str):    """Update a single document's embedding"""        # Generate new embedding    embedding = openai.Embedding.create(        input=new_content,        model="text-embedding-3-small"    )['data'][0]['embedding']        # Update in database    conn.execute("""        UPDATE documents        SET content = ?,            embedding = ?,            updated_at = CURRENT_TIMESTAMP        WHERE id = ?    """, [new_content, embedding, doc_id])        print(f"✅ Updated document: {doc_id}")def delete_document(doc_id: str):    """Remove a document"""    conn.execute("DELETE FROM documents WHERE id = ?", [doc_id])    print(f"🗑️ Deleted document: {doc_id}")# Example: Update changed docschanged_docs = [    ("quickstart_chunk_0", "Updated quickstart content..."),    ("api_auth_chunk_1", "New authentication method...")]for doc_id, content in changed_docs:    update_document(doc_id, content)
```

## Troubleshooting

### Issue: Poor search quality

**Problem:** Irrelevant results returned

**Solutions:**

1.  Increase `min_similarity` threshold (0.7 → 0.8)
2.  Use hybrid search (add keyword filters)
3.  Improve chunking strategy (smaller/larger chunks)
4.  Try different embedding model (text-3-large)

### Issue: Slow searches

**Problem:** Queries take > 50ms

**Solutions:**

1.  Ensure HNSW index is created: `CREATE INDEX USING HNSW`
2.  Add similarity threshold in WHERE clause
3.  Reduce limit (don't retrieve 100 results)
4.  Use smaller embedding dimension (768 vs 1536)

### Issue: LLM hallucinations

**Problem:** AI makes up answers not in docs

**Solutions:**

1.  Lower LLM temperature (0.3 or lower)
2.  Improve system prompt ("Only use provided context")
3.  Increase similarity threshold (fewer but better docs)
4.  Add explicit "I don't know" instruction

## Next Steps

*   **[Vector Search Guide](/docs/guides/connections/vector-search.md)**: Deep dive into vector search
*   **[MCP Overview](/docs/ai-integration/mcp-overview.md)**: AI agent integration
*   **[MCP Protocol](/docs/ai-integration/mcp-protocol.md)**: How tools, resources, and prompts fit together
*   **[MCP Prompts recipe](/docs/examples/mcp-prompts.md)**: More patterns for shipping prompts alongside your API
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Cache popular queries
*   **[Authentication](/docs/endpoints/authentication.md)**: Secure your API

## Complete Code

All code for this example is available in the [flAPI Examples Repository](https://github.com/datazoode/flapi-examples/tree/main/vector-search-rag).

---

**Production Tips:**

1.  **Cache embeddings** - Don't regenerate on every query
2.  **Monitor relevance** - Track similarity scores to tune thresholds
3.  **A/B test prompts** - Different system prompts yield different quality
4.  **Log queries** - Understand what users ask to improve docs
5.  **Version embeddings** - When switching models, keep old embeddings during transition

([🍪 Cookie Settings](#cookie-settings))

# Building from Source

## Prerequisites

*   C++ compiler with C++17 support
*   CMake 3.15 or higher
*   Ninja build system (recommended)
*   Git
*   vcpkg package manager

On Ubuntu/Debian, you can install the basic requirements with:

```
sudo apt-get updatesudo apt-get install -y build-essential cmake ninja-build
```

## Build Steps

1.  Clone the repository with submodules:

```
git clone --recurse-submodules https://github.com/datazoode/flapi.gitcd flapi
```

2.  Initialize DuckDB submodule (pinned at v1.5.2, see [CMakeLists.txt](https://github.com/datazoode/flapi/blob/main/CMakeLists.txt) `DUCKDB_EXPLICIT_VERSION`):

```
git submodule update --init --recursivecd duckdbgit checkout v1.5.2cd ..
```

3.  Set up vcpkg:

```
# Bootstrap vcpkg if you haven't already./vcpkg/bootstrap-vcpkg.sh# Integrate vcpkg with your system./vcpkg/vcpkg integrate install
```

4.  Build the project:

```
# Create build directorymkdir -p build/releasecd build/release# Configure with CMakecmake ../.. -G Ninja \    -DCMAKE_BUILD_TYPE=Release \    -DCMAKE_TOOLCHAIN_FILE=../../vcpkg/scripts/buildsystems/vcpkg.cmake# Buildninja# The binary will be available as 'flapi' in the current directory
```

## Verifying the Build

After building, you can verify the installation:

```
# Print the usage banner to confirm the binary runs./flapi --help# Validate a configuration without starting the server./flapi --validate-config -c examples/flapi.yaml# Optional: Move to system path for global accesssudo cp flapi /usr/local/bin/
```

You can also use the convenience make targets from the repository root:

```
make run-debug      # Runs build/debug/flapi with examples/flapi.yaml at --log-level debugmake run-release    # Runs build/release/flapi with examples/flapi.yaml at --log-level info
```

## Common Build Issues

1.  **Missing Dependencies**: If you encounter missing dependency errors, make sure vcpkg is properly initialized and integrated.
    
2.  **CMake Configuration Fails**: Ensure you have the correct version of CMake and all required development tools installed.
    
3.  **Build Memory Issues**: If the build process fails due to memory constraints, you can limit parallel jobs:
    

```
ninja -j2  # Limit to 2 parallel jobs
```

4.  **vcpkg Cache**: To speed up future builds, vcpkg caches packages. The cache is typically located in `~/.cache/vcpkg`.

## Development Setup

For development, you might want to build with debug symbols:

```
mkdir -p build/debugcd build/debugcmake ../.. -G Ninja \    -DCMAKE_BUILD_TYPE=Debug \    -DCMAKE_TOOLCHAIN_FILE=../../vcpkg/scripts/buildsystems/vcpkg.cmakeninja
```

The debug build will include additional information useful for development and debugging.

## Next Steps

*   **[Quickstart Guide](/docs/getting-started/quickstart.md)**: Build your first API
*   **[Configuration](/docs/getting-started/configuration.md)**: Learn all configuration options
*   **[Contributing](/docs/getting-started/contributing.md)**: Learn how to contribute to flAPI
*   **[CLI Tools](/docs/tools/cli-overview.md)**: Use the flapii CLI for development

([🍪 Cookie Settings](#cookie-settings))

# Configuration

flAPI uses YAML files for configuration. The main configuration file (`flapi.yaml`) defines global settings, connections, and server behaviour. This page is an introduction to each top-level section — see the upstream [Configuration Reference](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md) for the full option list.

## Naming Conventions

flAPI configuration keys use **hyphenated** names (`project-name`, `template-source`, `url-path`, `enforce-https`). The only exception is the `duckdb:` block, whose keys (`db_path`, `access_mode`, `max_memory`, `default_order`, `threads`, …) keep `snake_case` because they are forwarded verbatim to DuckDB.

## Project Metadata

```
project-name: example-flapi-projectproject-description: An example flAPI projecttemplate:  path: './sqls'            # Path to SQL templates and endpoint configs  environment-whitelist:    # Optional: which env vars may be substituted in templates    - '^FLAPI_.*'
```

Other top-level metadata keys: `server-name` (default `"localhost"`) and `http-port` (default `8080`).

See: [CONFIG\_REFERENCE.md §2.1 Project Metadata](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#21-project-metadata).

## DuckDB Settings

Configure the embedded DuckDB instance. Note the `snake_case` keys — they are passed through directly to DuckDB:

```
duckdb:  db_path: ./flapi_cache.db  # Optional: omit for in-memory (default :memory:)  access_mode: READ_WRITE    # READ_WRITE or READ_ONLY  threads: 8                 # Default: auto  max_memory: 8GB            # Default: auto  default_order: DESC        # ASC or DESC
```

Any additional key under `duckdb:` is forwarded as a DuckDB SET option.

See: [CONFIG\_REFERENCE.md §2.4 DuckDB Configuration](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#24-duckdb-configuration).

## Connections

Define your data source connections. Each connection is keyed by a unique name and may have an `init:` block (SQL run once when the connection is opened) and a `properties:` block (key-value pairs accessible in templates as `{{ conn.<key> }}`).

```
connections:  # Local Parquet file  customers-parquet:    properties:      path: './data/customers.parquet'  # BigQuery connection  bigquery-lakehouse:    init: |      INSTALL 'bigquery' FROM community;      LOAD 'bigquery';    properties:      project_id: 'my-project-id'
```

Default: no connections defined. See: [CONFIG\_REFERENCE.md §2.3 Connections](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#23-connections).

## DuckLake (Snapshot Cache)

DuckLake adds snapshot-based caching with retention, compaction, and a scheduler for periodic refreshes. It is disabled by default.

```
ducklake:  enabled: true  alias: cache                         # Catalog alias (default: "cache")  metadata-path: ./data/cache.ducklake # Metadata directory  data-path: ./data/cache              # Data files directory  retention:    keep-last-snapshots: 10    max-snapshot-age: 30d  compaction:    enabled: true    schedule: '@daily'  scheduler:    enabled: true    scan-interval: 5m                  # How often to check for scheduled refreshes
```

Default: `ducklake.enabled: false`. See: [CONFIG\_REFERENCE.md §2.5 DuckLake Configuration](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#25-ducklake-configuration).

## MCP (Model Context Protocol)

flAPI exposes both REST endpoints and MCP tools/resources/prompts. The MCP server is **enabled by default on port 8081**; you only need an `mcp:` block to override defaults or to provide LLM-facing instructions.

```
mcp:  enabled: true                              # Default: true  port: 8081                                 # Default: 8081  host: localhost  allow-list-changed-notifications: true  instructions-file: ./mcp_instructions.md   # Or inline via `instructions: |`
```

Per-endpoint MCP behaviour is configured inside endpoint YAML files (`mcp-tool:`, `mcp-resource:`, `mcp-prompt:`).

See: [CONFIG\_REFERENCE.md §2.6 MCP Configuration](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#26-mcp-configuration).

## Global Authentication

Set a default authentication policy that applies to all endpoints unless overridden at the endpoint level. Supported types: `basic`, `jwt`, `bearer`, `oidc`.

```
auth:  enabled: true  type: basic  users:    - username: admin      password: '{{env.ADMIN_PASSWORD}}'      roles: [admin, read, write]
```

```
auth:  enabled: true  type: bearer  jwt-secret: '{{env.JWT_SECRET}}'  jwt-issuer: '{{env.JWT_ISSUER}}'
```

Default: `auth.enabled: false`. See: [CONFIG\_REFERENCE.md §2.7 Authentication (Global)](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#27-authentication-global) and §7 for the per-type options.

## Global Rate Limiting

Apply a global request budget across all endpoints. Per-endpoint `rate-limit:` blocks override this.

```
rate_limit:  enabled: true  max: 100              # Default: 100  interval: 60          # Time window in seconds (default: 60)  key: user-or-ip       # Bucket strategy (default: ip)
```

`rate-limit.key` picks the bucket strategy:

*   `ip` (default, legacy) — per-IP. Behaves identically to older flAPI releases.
*   `user` — per-authenticated-principal. Unauthenticated requests are not rate-limited via this strategy (they fall back to the IP bucket if also configured).
*   `user-or-ip` — authenticated principal when present, IP fallback otherwise. **Recommended for share-NAT scenarios** where many users share a single egress IP.

Default: `rate_limit.enabled: false`. See: [CONFIG\_REFERENCE.md §2.8 Rate Limiting (Global)](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#28-rate-limiting-global).

## CORS Allow-List

flAPI no longer emits the wildcard `Access-Control-Allow-Origin: *` by default. Opt into specific origins via:

```
cors:  allow-origins:    - https://app.example.com    - https://staging.example.com
```

`flapii project init` still ships `["*"]` so first-run demos work; the startup auditor warns at boot when `*` is combined with `auth.enabled: true` (credential-bearing requests across origins).

## Request Audit Log

Per-request JSONL audit log for both REST and MCP traffic. Off by default — opt in:

```
audit:  enabled: true  sink: file            # `stdout` for container log collectors, or `file`  path: ./logs/audit.jsonl  redact:               # Parameter names replaced with <redacted>    - password    - api_key
```

Each event is a single JSON line:

```
{"ts":"2026-05-17T05:32:11Z","request_id":"…","principal":"alice","method":"tools/call","target":"customer_lookup","params":{"id":"42"},"status":"ok","row_count":1,"latency_ms":12}
```

## HTTPS Enforcement

flAPI can terminate TLS directly via OpenSSL. Reverse-proxy termination is still recommended for production, but direct TLS is supported for self-contained deployments.

```
# Bind the listener as HTTPS:https:  enabled: true  ssl_cert_file: ./ssl/cert.pem  ssl_key_file: ./ssl/key.pem# Optional: redirect plain-HTTP requests to HTTPS instead of refusing themenforce-https:  enabled: true
```

See: [CONFIG\_REFERENCE.md §2.9 HTTPS Configuration](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#29-https-configuration).

## Startup Security Auditor

At boot, flAPI scans the loaded config and emits structured warnings for:

*   Plaintext passwords in `auth.users[*].password` — recommend migrating to `$pbkdf2-sha256$…` MCF hashes.
*   MD5 password hashes (32-char hex) — recommend migrating to PBKDF2-SHA256.
*   MCP exposed without auth on a non-loopback bind — large blast radius for a misconfigured agent.
*   CORS wildcard combined with `auth.enabled: true` — credential-bearing requests across origins.

The warnings appear at INFO level in the startup logs; the server still starts. To opt into stricter behaviour, combine with `mcp.strict-descriptions: true` (refuses to start if any MCP tool description fails the prompt-injection hygiene scan).

## Heartbeat

A background worker that emits periodic health signals.

```
heartbeat:  enabled: true            # Default: false  worker-interval: 10      # Seconds (default: 60)
```

See: [CONFIG\_REFERENCE.md §2.10 Heartbeat Configuration](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#210-heartbeat-configuration).

## Remote Configuration & Templates (VFS)

flAPI can load `flapi.yaml` itself, the SQL templates directory, and individual `template-source:` files from remote storage via DuckDB's Virtual File System. Supported schemes: `https://`, `http://`, `s3://`, `s3a://`, `s3n://`, `gs://`, `az://`, `abfs://`, `file://` (defaults: `file` and `https` only — additional schemes are enabled by loading the relevant DuckDB extension in a connection's `init:` block).

```
# Load main config from HTTPS or S3./flapi --config https://example.com/configs/flapi.yaml./flapi --config s3://my-bucket/configs/flapi.yaml
```

```
# Or point the template root at a remote locationtemplate:  path: s3://my-bucket/templates/
```

Cloud-storage credentials are read from the usual environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `GOOGLE_APPLICATION_CREDENTIALS`, `AZURE_STORAGE_ACCOUNT`, `AZURE_STORAGE_KEY`, …).

See: [CONFIG\_REFERENCE.md §2.11 Storage Configuration (VFS)](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#211-storage-configuration-vfs).

## Telemetry

flAPI sends anonymous `application_start` / `application_stop` events to PostHog by default. To opt out:

```
telemetry:  enabled: false   # Default: true
```

Opt-out precedence (highest wins): `--no-telemetry` CLI flag → `FLAPI_NO_TELEMETRY=1` → `telemetry.enabled: false` → `DATAZOO_DISABLE_TELEMETRY=1`.

See: [CONFIG\_REFERENCE.md §2.12 Telemetry Configuration](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md#212-telemetry-configuration).

## Environment Variables in Configuration

You can substitute environment variables anywhere in YAML using the `${VAR_NAME}` syntax. Only variables matching the patterns in `template.environment-whitelist` are exposed.

```
template:  environment-whitelist:    - '^FLAPI_.*'    - '^DB_.*'connections:  postgres-db:    properties:      host: '${DB_HOST}'      password: '${DB_PASSWORD}'
```

## Complete Example

```
project-name: example-flapi-projectproject-description: An example flAPI projecttemplate:  path: './sqls'  environment-whitelist:    - '^FLAPI_.*'connections:  customers-parquet:    properties:      path: './data/customers.parquet'duckdb:  db_path: ./flapi_cache.db  access_mode: READ_WRITE  threads: 8  max_memory: 8GB  default_order: DESCducklake:  enabled: true  alias: cache  metadata-path: ./data/cache.ducklake  data-path: ./data/cache  retention:    keep-last-snapshots: 10    max-snapshot-age: 30dmcp:  enabled: true  port: 8081  host: localhostauth:  enabled: falserate_limit:  enabled: true  max: 100  interval: 60heartbeat:  enabled: true  worker-interval: 10enforce-https:  enabled: falsetelemetry:  enabled: true
```

## Next Steps

*   **[Create Your First API](/docs/getting-started/first-api.md)**: Build a complete API with validation
*   **[Quickstart Guide](/docs/getting-started/quickstart.md)**: Build your first API in 5 minutes
*   **[Connect Data Sources](/docs/guides/connections/overview.md)**: Configure BigQuery, PostgreSQL, Parquet, and more
*   **[Endpoint Configuration](/docs/endpoints/overview.md)**: Define REST API endpoints
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Create dynamic SQL queries
*   **[Deployment Guide](/docs/getting-started/deployment.md)**: Deploy flAPI to production

([🍪 Cookie Settings](#cookie-settings))

# Contributing

We welcome contributions to Flapi! Here's how you can help:

## Filing Issues

If you find a bug or have a feature request, please file an issue on our GitHub repository.

## Pull Requests

1.  Fork the repository
2.  Create your feature branch
3.  Submit a pull request

## Development Setup

For local development:

1.  **[Build from Source](/docs/getting-started/build-from-source.md)**: Compile flAPI locally
2.  **[CLI Tools](/docs/tools/cli-overview.md)**: Use the flapii CLI for testing
3.  **[VS Code Extension](/docs/tools/vscode-extension.md)**: Integrated development experience

## Documentation

Help improve documentation:

*   All documentation is in the `docs/` directory
*   Written in Markdown with Docusaurus
*   Submit documentation improvements via pull requests

## Next Steps

*   **[GitHub Repository](https://github.com/datazoode/flapi)**: View source code
*   **[GitHub Discussions](https://github.com/datazoode/flapi/discussions)**: Ask questions
*   **[Build from Source](/docs/getting-started/build-from-source.md)**: Compile and develop locally

([🍪 Cookie Settings](#cookie-settings))

# Deployment

**Work in Progress:**

This deployment documentation is currently being enhanced and refined. Some sections may be incomplete or subject to change. We're actively working on comprehensive examples and troubleshooting guides for all platforms.

flAPI is designed for deployment flexibility. Available as a single binary or Docker container, it runs on any platform that supports Linux containers or x86-64 binaries.

## Quick Start

### 🚀 Interactive Deployment Configurator

Configure your deployment with config mounting strategy

PlatformGoogle Cloud Run (Serverless Container)AWS Lambda (Serverless Function)Fly.io (Edge Platform)Railway (PaaS)Kubernetes (Container Orchestration)IONOS Cloud (European Cloud)Hetzner Cloud (VPS)

Configuration StrategyLocal Volume Mount

Mount config from host filesystem

Regionnbg1fsn1hel1

Memory1 GB2 GB4 GB8 GB

CPU1 vCPU2 vCPU4 vCPU

Environment Variables (optional)

Generated Configuration📋 Copy

```
# 1. Create directory structure on host
mkdir -p ~/flapi/{config/sqls,data}

# 2. Create your flapi.yaml
cat > ~/flapi/config/flapi.yaml <<EOF
project_name: my-api
template:
  path: '/config/sqls'
connections:
  # Your connections here
duckdb:
  db_path: /data/flapi.db
EOF

# 3. Add your SQL templates
# cp your_templates.yaml ~/flapi/config/sqls/

# 4. Create docker-compose.yml
cat > ~/flapi/docker-compose.yml <<EOF
version: '3.8'
services:
  flapi:
    image: ghcr.io/datazoode/flapi:latest
    ports:
      - "8080:8080"
    volumes:
      - ./config:/config:ro
      - ./data:/data
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2G
EOF

# 5. Deploy
cd ~/flapi && docker-compose up -d
```

💡 Tip: Your config must include flapi.yaml and SQL templates in /config directory

## Requirements

| Resource | Minimum | Recommended | Notes |
| --- | --- | --- | --- |
| **CPU** | 1 vCPU | 2 vCPU | DuckDB is CPU-intensive for aggregations |
| **Memory** | 1 GB | 2-4 GB | Depends on cache size |
| **Storage** | 1 GB | 10 GB+ | For DuckDB cache persistence |
| **Port** | 8080 | \-  | Default HTTP port |
| **Startup** | < 100ms | \-  | Fast cold starts for serverless |

## Configuration Management

**Critical:**

**flAPI requires configuration files to function**. The base Docker image (`ghcr.io/datazoode/flapi:latest`) contains only the binary - you must mount your configuration to run it.

### Standard Directory Structure

flAPI expects the following structure:

```
/config/├── flapi.yaml           # Main configuration file (required)└── sqls/                # SQL template directory (required)    ├── users.yaml       # Endpoint definitions    ├── products.yaml    └── orders.yaml
```

### Minimal Example Configuration

Create `flapi.yaml`:

config/flapi.yaml

```
project-name: my-apiproject-description: My flAPI deploymenttemplate:  path: '/config/sqls'  # Path to SQL templates and endpoint configsconnections:  postgres:    init: |      INSTALL postgres;      LOAD postgres;      ATTACH 'host=db.example.com port=5432 dbname=mydb user=app password=${DB_PASSWORD}'        AS pgdb (TYPE postgres);    properties:      schema: publicduckdb:  db_path: /data/flapi.db  # Cache database location  access_mode: READ_WRITE
```

Create your first endpoint `sqls/users.yaml` and its SQL template `sqls/users.sql`. Endpoint configs always reference an external SQL file via `template-source:` — there is no inline `sql:` field.

config/sqls/users.yaml

```
url-path: /usersmethod: GETtemplate-source: users.sqlconnection:  - postgres
```

config/sqls/users.sql

```
SELECT *FROM pgdb.usersWHERE active = trueLIMIT 100
```

### Configuration Mounting Strategies

Choose the approach that fits your deployment platform:

#### 1️⃣ Local Volume Mount (Docker, VPS, K8s)

**Best for**: Development, VPS deployment, on-premises

Mount your local config directory directly into the container:

```
docker run -d \  -p 8080:8080 \  -v $(pwd)/config:/config:ro \  -v $(pwd)/data:/data \ghcr.io/datazoode/flapi:latest
```

**Advantages**:

*   ✅ Simple and direct
*   ✅ Easy to update (edit files, restart container)
*   ✅ Works with any text editor

**Disadvantages**:

*   ❌ Requires host filesystem access
*   ❌ Not suitable for serverless platforms

#### 2️⃣ Kubernetes ConfigMap (K8s, IONOS, STACKIT)

**Best for**: Kubernetes deployments, multi-replica setups

Create ConfigMaps from your config files:

```
# Create ConfigMap for main configkubectl create configmap flapi-config \  --from-file=flapi.yaml=./config/flapi.yaml \  --namespace=flapi# Create ConfigMap for SQL templateskubectl create configmap flapi-sqls \  --from-file=./config/sqls/ \  --namespace=flapi
```

Mount in your Deployment:

```
volumeMounts:- name: config  mountPath: /config/flapi.yaml  subPath: flapi.yaml- name: sqls  mountPath: /config/sqlsvolumes:- name: config  configMap:    name: flapi-config- name: sqls  configMap:    name: flapi-sqls
```

**Advantages**:

*   ✅ Native Kubernetes integration
*   ✅ Version controlled via kubectl
*   ✅ Can be updated without rebuilding images

**Disadvantages**:

*   ❌ ConfigMap size limit (1MB)
*   ❌ Requires kubectl access to update

**Tip:**

For large SQL template collections, use PersistentVolumes instead of ConfigMaps.

#### 3️⃣ Cloud Storage Mount (Cloud Run, Lambda, Azure)

**Best for**: Serverless platforms, multi-region deployments

**Google Cloud Run (GCS Bucket)**:

```
# 1. Upload configgsutil -m cp -r ./config gs://my-bucket/flapi-config/# 2. Deploy with GCS mountgcloud run deploy flapi \  --image ghcr.io/datazoode/flapi:latest \  --add-volume=name=config,type=cloud-storage,bucket=my-bucket \  --add-volume-mount=volume=config,mount-path=/config
```

**AWS Lambda (EFS Mount)**:

```
# 1. Create EFS file systemaws efs create-file-system --tags Key=Name,Value=flapi-config# 2. Mount EFS to EC2, upload config to /config# 3. Create Lambda with EFS mountaws lambda create-function \  --function-name flapi \  --file-system-configs "Arn=arn:aws:elasticfilesystem:...,LocalMountPath=/config"
```

**Advantages**:

*   ✅ Works with serverless platforms
*   ✅ Centralized configuration storage
*   ✅ Can be shared across multiple instances

**Disadvantages**:

*   ❌ Increased cold start time
*   ❌ Additional cost for storage
*   ❌ More complex setup

#### 5️⃣ Self-Packaged Binary (Single-File Deploy)

**Best for**: `scp` deploys, container `FROM scratch` images, air-gapped environments, immutable distribution

The `flapi` binary can fold an entire config tree (`flapi.yaml`, endpoint YAMLs, SQL templates, small data files) into itself, producing one self-contained executable. The same binary that serves the API is the packager.

```
# 1. Pack a config tree into a new bundled binaryflapi pack --in ./config --out flapi-prod# 2. Inspect what's bundled (EOCD offset, entry list, byte counts)./flapi-prod info# 3. (Optional) extract the bundle for debugging or audit./flapi-prod unpack --to /tmp/extracted# 4. Deploy — one file, no config-tree mount, no Docker requiredscp flapi-prod user@host:/opt/flapi/ssh user@host '/opt/flapi/flapi-prod'    # serves bundled config from any cwd
```

**How it works under the hood:**

*   A ZIP archive is appended after the executable on Linux/Windows (or written into a pre-allocated `__FLAPI/__bundle` Mach-O segment on macOS, then re-`codesign`\-ed so the output is notarisable).
*   At startup, flAPI reverse-scans for the bundle (or probes the segment on macOS) and registers an `EmbeddedArchiveFileProvider` plus an `embed://` DuckDB filesystem so config / SQL templates / `read_csv()` calls all resolve to the in-memory bundle.
*   If no bundle is present, all code paths fall back to the local filesystem unchanged — existing operators see no behaviour change.

**Reproducible builds:**

```
SOURCE_DATE_EPOCH=1700000000 flapi pack --in config --out aSOURCE_DATE_EPOCH=1700000000 flapi pack --in config --out bsha256sum a b   # identical
```

**Advantages**:

*   ✅ One artifact deploys; no config-tree mount, ConfigMap, or volume needed
*   ✅ Works on `scratch` Docker images, air-gapped hosts, lambda/cf bundles
*   ✅ Reproducible builds (byte-identical with `SOURCE_DATE_EPOCH`)
*   ✅ macOS notarisable via reserved-segment + re-codesign
*   ✅ Bundle is immutable — easy supply-chain story

**Disadvantages**:

*   ❌ Bundled binaries are immutable; live config edits require a rebuild
*   ❌ Reserved-segment size on macOS is fixed at link time (16 MiB default; rebuild with larger `FLAPI_RESERVED_BUNDLE_MIB` for bigger trees)
*   ❌ Best for static config; less suited to per-tenant overrides

**Secrets stay out of the bundle:**

`flapi pack` refuses files matching `*.env`, `secrets/*`, `*.pem`, `*.key` by default (a bundled binary that contains a password _is_ a secret). Credentials come from the environment at runtime: `AWS_*`, `GOOGLE_*`, `AZURE_*`, `FLAPI_CONFIG_SERVICE_TOKEN`, and `{{env.VAR}}` interpolation inside whitelisted YAML.

See [Server CLI § Self-packaging subcommands](/docs/tools/server-cli.md#self-packaging-subcommands) for full reference.

#### 4️⃣ Baked-In Configuration (Custom Image)

**Best for**: Immutable deployments, CI/CD pipelines

Create a custom Docker image with config baked in:

Dockerfile

```
FROM ghcr.io/datazoode/flapi:latest# Copy configuration into imageCOPY config/flapi.yaml /config/flapi.yamlCOPY config/sqls /config/sqls# The config-file path is passed via CLI (-c), not via env varCMD ["/app/flapi", "-c", "/config/flapi.yaml"]
```

Build and deploy:

```
docker build -t my-org/flapi:v1.0 .docker push my-org/flapi:v1.0
```

**Advantages**:

*   ✅ Self-contained, portable image
*   ✅ Fastest cold start (no external mounts)
*   ✅ Works on any platform

**Disadvantages**:

*   ❌ Config changes require image rebuild
*   ❌ Larger image size
*   ❌ Less flexible

**Security:**

Never bake secrets (database passwords, API keys) into images. Use environment variables or secret management instead.

### Configuration Update Strategies

| Platform | Update Method | Downtime | Rollback |
| --- | --- | --- | --- |
| **Docker Compose** | Edit files → `docker-compose restart` | < 1s | Manual (git revert) |
| **Kubernetes** | `kubectl apply configmap` → Rolling update | 0s  | `kubectl rollout undo` |
| **Cloud Run** | Update GCS bucket → Redeploy | ~30s | Deploy previous revision |
| **Custom Image** | Rebuild → Push → Deploy | ~2-5min | Deploy previous tag |

### Configuration Validation

Before deploying, validate your configuration:

```
# Validate configuration and exit (does NOT start the server)docker run --rm \  -v $(pwd)/config:/config \  ghcr.io/datazoode/flapi:latest \  -c /config/flapi.yaml --validate-config# Check YAML syntaxyamllint config/flapi.yaml config/sqls/*.yaml# Test SQL templates# (flAPI will validate SQL on startup)
```

### Environment Variables

The flAPI server itself does not read its config-file path, HTTP port, or log level from environment variables — those are set via CLI flags (`-c`, `-p`, `--log-level`). What environment variables _do_ control:

| Variable | Read by | Purpose |
| --- | --- | --- |
| `FLAPI_CONFIG` | `flapi` server | Path to `flapi.yaml` (fallback for `-c` / `--config`). Precedence: CLI > env > built-in default `flapi.yaml` |
| `FLAPI_LOG_LEVEL` | `flapi` server | Log verbosity (fallback for `--log-level`). Invalid values exit non-zero. Precedence: CLI > env > `info` |
| `FLAPI_CONFIG_SERVICE_TOKEN` | `flapi` server | Bearer token for the runtime config service API (fallback for `--config-service-token`) |
| `FLAPI_NO_TELEMETRY` | `flapi` server | Set to `1`, `true`, or `yes` to disable startup/shutdown telemetry |
| `SOURCE_DATE_EPOCH` | `flapi pack` | Mtime stamped on every bundle entry → reproducible (byte-identical) pack output |
| `CODESIGN_IDENTITY` | `flapi pack` (macOS only) | Identity passed to `codesign --sign` after packing. Defaults to `-` (ad-hoc) |
| `FLAPI_BASE_URL` | `flapii` CLI client | Default base URL for the config service |
| `FLAPI_TOKEN` | `flapii` CLI client | Bearer token used by `flapii` |
| `FLAPI_TIMEOUT` | `flapii` CLI client | Request timeout (seconds) |
| `FLAPI_RETRIES` | `flapii` CLI client | Number of retry attempts |
| `FLAPI_GEMINI_KEY` | `flapii` CLI client | Gemini API key for AI-assisted features |

Beyond these, any environment variable whose name matches a regex in `template.environment-whitelist` may be substituted inside `flapi.yaml` and SQL templates via `${VAR_NAME}` — that is how secrets like `DB_PASSWORD` reach the running server.

```
# Docker — pass secrets as substitution sources, not as flAPI-specific overridesdocker run \  -e DB_PASSWORD="${DB_PASSWORD}" \  -e FLAPI_NO_TELEMETRY=1 \  -v $(pwd)/config:/config:ro \  -p 8080:8080 \  ghcr.io/datazoode/flapi:latest \  -c /config/flapi.yaml -p 8080 --log-level info# Kubernetesenv:- name: DB_PASSWORD  valueFrom:    secretKeyRef:      name: db-secrets      key: password- name: FLAPI_NO_TELEMETRY  value: "1"
```

> Valid `--log-level` values: `debug`, `info`, `warning`, `error` (note: `warning`, not `warn`).

## Enterprise Cloud Platforms

### Google Cloud Run

**Best for**: Serverless container deployment with auto-scaling to zero

```
# Deploy to Cloud Run.# The config-file path can be passed as a CLI arg (--args=...) or via# the FLAPI_CONFIG env var (CLI wins when both are set).gcloud run deploy flapi \  --image ghcr.io/datazoode/flapi:latest \  --platform managed \  --region europe-west1 \  --port 8080 \  --memory 2Gi \  --cpu 1 \  --allow-unauthenticated \  --args="-c,/config/flapi.yaml"
```

**Pricing**: Pay per request, scales to zero

*   **Free tier**: 2 million requests/month
*   **Cost**: ~$0.00002400 per request (beyond free tier)

### AWS App Runner

**Best for**: Fully managed container service with simple deployment

```
# Create apprunner.yamlversion: 1.0runtime: python3build:  commands:    image: ghcr.io/datazoode/flapi:latestrun:  runtime-version: 3.8  command: /app/flapi -c /config/flapi.yaml  network:    port: 8080  env:    - name: FLAPI_ENV      value: production# Deployaws apprunner create-service \  --service-name flapi \  --source-configuration file://apprunner.yaml
```

**Pricing**: Pay for active time

*   **Cost**: ~$0.064/vCPU-hour + $0.007/GB-hour

### AWS Lambda

**Best for**: Sporadic traffic patterns, event-driven workloads

```
# Package as container imagedocker build -t flapi-lambda .# Push to ECRaws ecr get-login-password --region us-east-1 | \  docker login --username AWS --password-stdin ACCOUNT.dkr.ecr.us-east-1.amazonaws.comdocker push ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/flapi:latest# Create function. Use the image's CMD (set in your Dockerfile) to point# at the config file with `-c /config/flapi.yaml`; flAPI does not read a# FLAPI_CONFIG env var.aws lambda create-function \  --function-name flapi \  --package-type Image \  --code ImageUri=ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/flapi:latest \  --role arn:aws:iam::ACCOUNT:role/lambda-flapi \  --memory-size 2048 \  --timeout 30
```

**Cache Persistence:**

Lambda containers are ephemeral. Use EFS for persistent DuckDB cache:

```
--file-system-config Arn=arn:aws:elasticfilesystem:REGION:ACCOUNT:access-point/ACCESS_POINT
```

**Pricing**: Pay per invocation

*   **Free tier**: 1M requests + 400,000 GB-seconds/month
*   **Cost**: $0.20 per 1M requests + $0.0000166667 per GB-second

### Azure Container Apps

**Best for**: Kubernetes-based microservices without cluster management

```
az containerapp create \  --name flapi \  --resource-group myResourceGroup \  --image ghcr.io/datazoode/flapi:latest \  --target-port 8080 \  --ingress external \  --min-replicas 1 \  --max-replicas 10 \  --cpu 1.0 \  --memory 2.0Gi \  --command "/app/flapi" --args "-c,/config/flapi.yaml"
```

**Pricing**: Pay for vCPU and memory

*   **Cost**: ~$0.000012/vCPU-second + $0.000002/GB-second

## European Cloud Providers

### IONOS Cloud

**Best for**: German/European data sovereignty, GDPR compliance

IONOS provides managed Kubernetes and container hosting with data centers in Germany and Spain.

```
# Install IONOS CLIcurl -sL https://github.com/ionos-cloud/ionosctl/releases/latest/download/ionosctl-linux-amd64.tar.gz | tar -xzv# Create Kubernetes cluster (if not exists)ionosctl k8s cluster create \  --name flapi-cluster \  --k8s-version 1.28 \  --datacenter-id DATACENTER_ID# Deploy flAPIkubectl apply -f - <<EOFapiVersion: apps/v1kind: Deploymentmetadata:  name: flapi  labels:    app: flapispec:  replicas: 2  selector:    matchLabels:      app: flapi  template:    metadata:      labels:        app: flapi    spec:      containers:      - name: flapi        image: ghcr.io/datazoode/flapi:latest        ports:        - containerPort: 8080        resources:          requests:            memory: "2Gi"            cpu: "1"          limits:            memory: "4Gi"            cpu: "2"        volumeMounts:        - name: config          mountPath: /config        - name: data          mountPath: /data      volumes:      - name: config        persistentVolumeClaim:          claimName: flapi-config      - name: data        persistentVolumeClaim:          claimName: flapi-data---apiVersion: v1kind: Servicemetadata:  name: flapispec:  type: LoadBalancer  ports:  - port: 80    targetPort: 8080  selector:    app: flapiEOF
```

**Key Features**:

*   Data centers: Frankfurt (de/fra), Berlin (de/txl), Logroño (es/vit)
*   GDPR-compliant infrastructure
*   ISO 27001 certified
*   German support

**Pricing**: Managed Kubernetes

*   **Cost**: ~€0.10/hour per worker node + €0.01/GB storage

### STACKIT

**Best for**: Schwarz Group ecosystem, German cloud sovereignty

```
# Create STACKIT Kubernetes cluster via portal or CLIstackit kubernetes cluster create \  --project-id PROJECT_ID \  --name flapi-cluster# Get kubeconfigstackit kubernetes credentials get \  --cluster-id CLUSTER_ID \  --project-id PROJECT_ID# Deploy flAPIkubectl apply -f - <<EOFapiVersion: apps/v1kind: Deploymentmetadata:  name: flapispec:  replicas: 2  selector:    matchLabels:      app: flapi  template:    metadata:      labels:        app: flapi    spec:      containers:      - name: flapi        image: ghcr.io/datazoode/flapi:latest        ports:        - containerPort: 8080        resources:          requests:            memory: "2Gi"            cpu: "1000m"---apiVersion: v1kind: Servicemetadata:  name: flapispec:  type: LoadBalancer  ports:  - port: 80    targetPort: 8080  selector:    app: flapiEOF
```

**Key Features**:

*   German data centers (operated by Schwarz Digits)
*   Enterprise SLA
*   Direct integration with SAP systems
*   Carbon-neutral operations

## Kubernetes

### Generic Kubernetes Deployment

**Best for**: Multi-cloud portability, full control, enterprise scale

Create a complete flAPI deployment with persistent storage and horizontal scaling:

flapi-k8s.yaml

```
# NamespaceapiVersion: v1kind: Namespacemetadata:  name: flapi---# ConfigMap for flapi.yamlapiVersion: v1kind: ConfigMapmetadata:  name: flapi-config  namespace: flapidata:  flapi.yaml: |    project-name: production-api    project-description: Production flAPI deployment    template:      path: '/config/sqls'    connections:      # Your connections here    duckdb:      db_path: /data/flapi.db      access_mode: READ_WRITE---# PersistentVolumeClaim for DuckDB cacheapiVersion: v1kind: PersistentVolumeClaimmetadata:  name: flapi-data  namespace: flapispec:  accessModes:    - ReadWriteOnce  resources:    requests:      storage: 20Gi  storageClassName: standard---# DeploymentapiVersion: apps/v1kind: Deploymentmetadata:  name: flapi  namespace: flapi  labels:    app: flapispec:  replicas: 3  selector:    matchLabels:      app: flapi  template:    metadata:      labels:        app: flapi    spec:      containers:      - name: flapi        image: ghcr.io/datazoode/flapi:latest        ports:        - containerPort: 8080          name: http        resources:          requests:            memory: "2Gi"            cpu: "1"          limits:            memory: "4Gi"            cpu: "2"        volumeMounts:        - name: config          mountPath: /config        - name: data          mountPath: /data        env:        - name: FLAPI_ENV          value: "production"        livenessProbe:          httpGet:            path: /health            port: 8080          initialDelaySeconds: 10          periodSeconds: 30        readinessProbe:          httpGet:            path: /health            port: 8080          initialDelaySeconds: 5          periodSeconds: 10      volumes:      - name: config        configMap:          name: flapi-config      - name: data        persistentVolumeClaim:          claimName: flapi-data---# ServiceapiVersion: v1kind: Servicemetadata:  name: flapi  namespace: flapispec:  type: LoadBalancer  ports:  - port: 80    targetPort: 8080    protocol: TCP    name: http  selector:    app: flapi---# HorizontalPodAutoscalerapiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata:  name: flapi  namespace: flapispec:  scaleTargetRef:    apiVersion: apps/v1    kind: Deployment    name: flapi  minReplicas: 2  maxReplicas: 10  metrics:  - type: Resource    resource:      name: cpu      target:        type: Utilization        averageUtilization: 70  - type: Resource    resource:      name: memory      target:        type: Utilization        averageUtilization: 80
```

**Deploy:**

```
kubectl apply -f flapi-k8s.yaml
```

### Ingress Configuration

For production deployments, add an Ingress with TLS:

flapi-ingress.yaml

```
apiVersion: networking.k8s.io/v1kind: Ingressmetadata:  name: flapi  namespace: flapi  annotations:    cert-manager.io/cluster-issuer: "letsencrypt-prod"    nginx.ingress.kubernetes.io/ssl-redirect: "true"spec:  ingressClassName: nginx  tls:  - hosts:    - api.yourdomain.com    secretName: flapi-tls  rules:  - host: api.yourdomain.com    http:      paths:      - path: /        pathType: Prefix        backend:          service:            name: flapi            port:              number: 80
```

## Indie Hacker Platforms

### Fly.io

**Best for**: Edge deployment, global distribution, developer experience

```
# Install flyctlcurl -L https://fly.io/install.sh | sh# Launch flAPIfly launch \  --image ghcr.io/datazoode/flapi:latest \  --name flapi-app \  --region fra \  --vm-size shared-cpu-2x \  --vm-memory 2048# Add persistent volume for cachefly volumes create flapi_data --region fra --size 10# Update fly.toml for volumecat > fly.toml <<EOFapp = "flapi-app"[build]  image = "ghcr.io/datazoode/flapi:latest"[[mounts]]  source = "flapi_data"  destination = "/data"[[services]]  internal_port = 8080  protocol = "tcp"  [[services.ports]]    handlers = ["http"]    port = 80  [[services.ports]]    handlers = ["tls", "http"]    port = 443EOF# Deployfly deploy
```

**Pricing**: Free tier available

*   **Free**: 3 shared-cpu-1x VMs (256MB RAM)
*   **Paid**: ~$0.0000022/second for shared-cpu-2x (2GB RAM)

### Railway

**Best for**: Git-based deployment, simple setup, PostgreSQL integration

```
# Install Railway CLInpm i -g @railway/cli# Loginrailway login# Initialize projectrailway init# Deploy from Docker imagerailway up --image ghcr.io/datazoode/flapi:latest# Or use railway.json for configurationcat > railway.json <<EOF{  "build": {    "dockerImage": "ghcr.io/datazoode/flapi:latest"  },  "deploy": {    "startCommand": "/app/flapi -c /config/flapi.yaml",    "restartPolicyType": "ON_FAILURE",    "healthcheckPath": "/health",    "healthcheckTimeout": 100  }}EOF
```

**Pricing**: $5 credit/month on free plan

*   **Free**: $5/month credit (covers ~500 hours)
*   **Paid**: $0.000231/GB-hour RAM + $0.00463/vCPU-hour

### Render

**Best for**: Free tier for testing, managed services

```
# Create render.yamlcat > render.yaml <<EOFservices:  - type: web    name: flapi    env: docker    image:      url: ghcr.io/datazoode/flapi:latest    plan: starter    healthCheckPath: /health    envVars:      - key: FLAPI_ENV        value: production    disk:      name: flapi-data      mountPath: /data      sizeGB: 10EOF# Deploy via Render Dashboard or CLIrender deploy
```

**Pricing**: Free tier available

*   **Free**: 750 hours/month, sleeps after inactivity
*   **Starter**: $7/month (persistent, no sleep)
*   **Standard**: $25/month (2 GB RAM, 1 vCPU)

### DigitalOcean App Platform

**Best for**: Predictable pricing, managed databases, simple scaling

```
# Create .do/app.yamlmkdir -p .docat > .do/app.yaml <<EOFname: flapiservices:- name: api  image:    registry_type: GHCR    repository: datazoode/flapi    tag: latest  http_port: 8080  instance_count: 2  instance_size_slug: basic-xs  routes:  - path: /  health_check:    http_path: /health  envs:  - key: FLAPI_ENV    value: productionEOF# Deploy via doctldoctl apps create --spec .do/app.yaml# Or use DigitalOcean Dashboard
```

**Pricing**: Predictable monthly cost

*   **Basic (512MB)**: $5/month
*   **Basic (1GB)**: $12/month
*   **Professional (2GB)**: $24/month

## Cost-Effective VPS

### Hetzner Cloud

**Best for**: European hosting, best price/performance ratio

```
# Create server via CLIhcloud server create \  --name flapi \  --type cx21 \  --image docker-ce \  --ssh-key my-key# SSH into serverssh root@YOUR_SERVER_IP# Create docker-compose.ymlcat > docker-compose.yml <<EOFversion: '3.8'services:  flapi:    image: ghcr.io/datazoode/flapi:latest    container_name: flapi    restart: unless-stopped    ports:      - "8080:8080"    volumes:      - ./config:/config:ro      - ./data:/data      - ./sqls:/sqls:ro    environment:      - FLAPI_ENV=production    deploy:      resources:        limits:          cpus: '2'          memory: 4G        reservations:          cpus: '1'          memory: 2G    healthcheck:      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]      interval: 30s      timeout: 10s      retries: 3      start_period: 40s  # Optional: Caddy for HTTPS  caddy:    image: caddy:latest    container_name: caddy    restart: unless-stopped    ports:      - "80:80"      - "443:443"    volumes:      - ./Caddyfile:/etc/caddy/Caddyfile:ro      - caddy_data:/data      - caddy_config:/configvolumes:  caddy_data:  caddy_config:EOF# Create Caddyfile for HTTPScat > Caddyfile <<EOFapi.yourdomain.com {    reverse_proxy flapi:8080        # Enable compression    encode gzip        # Security headers    header {        Strict-Transport-Security "max-age=31536000;"        X-Content-Type-Options "nosniff"        X-Frame-Options "DENY"    }}EOF# Start servicesdocker-compose up -d# View logsdocker-compose logs -f flapi
```

**Pricing**: Excellent value

*   **CX11 (2GB)**: €4.15/month (~$4.50)
*   **CX21 (4GB)**: €5.83/month (~$6.35)
*   **CX31 (8GB)**: €10.59/month (~$11.50)

## Self-Hosted

### Systemd Service (Linux Binary)

For running flAPI as a native Linux service without Docker:

```
# Download flAPI binarycurl -L https://github.com/datazoode/flapi/releases/latest/download/flapi \  -o /usr/local/bin/flapichmod +x /usr/local/bin/flapi# Create flapi usersudo useradd -r -s /bin/false flapi# Create directoriessudo mkdir -p /etc/flapi/{sqls,config}sudo mkdir -p /var/lib/flapi/datasudo chown -R flapi:flapi /etc/flapi /var/lib/flapi# Create systemd servicesudo tee /etc/systemd/system/flapi.service > /dev/null <<EOF[Unit]Description=flAPI REST API ServiceDocumentation=https://flapi.ioAfter=network-online.targetWants=network-online.target[Service]Type=simpleUser=flapiGroup=flapiExecStart=/usr/local/bin/flapi -c /etc/flapi/flapi.yamlRestart=alwaysRestartSec=5StartLimitInterval=0StartLimitBurst=5WorkingDirectory=/var/lib/flapi# Security hardeningNoNewPrivileges=truePrivateTmp=trueProtectSystem=strictProtectHome=trueReadWritePaths=/var/lib/flapi# Resource limitsLimitNOFILE=65536MemoryLimit=4G[Install]WantedBy=multi-user.targetEOF# Copy your configurationsudo cp flapi.yaml /etc/flapi/sudo cp -r sqls/* /etc/flapi/sqls/sudo chown -R flapi:flapi /etc/flapi# Enable and startsudo systemctl daemon-reloadsudo systemctl enable flapisudo systemctl start flapi# Check statussudo systemctl status flapi# View logssudo journalctl -u flapi -f
```

## Production Checklist

Before deploying to production, ensure you have:

### Security

*   [ ]  Enable HTTPS/TLS (Let's Encrypt, Cloudflare, load balancer)
*   [ ]  Configure authentication (JWT, API keys)
*   [ ]  Set up rate limiting
*   [ ]  Whitelist environment variables in config
*   [ ]  Review row-level security in SQL templates
*   [ ]  Use secrets management (not hardcoded)

### Monitoring

*   [ ]  Configure health check endpoint (`/health`)
*   [ ]  Set up metrics collection (Prometheus, CloudWatch)
*   [ ]  Configure logging (structured JSON logs)
*   [ ]  Set up alerting (uptime, error rates, cache hits)
*   [ ]  Monitor cache refresh success/failures

### Performance

*   [ ]  Size memory for your cache needs
*   [ ]  Configure cache refresh schedules
*   [ ]  Set up horizontal scaling (HPA)
*   [ ]  Test under expected load
*   [ ]  Configure connection pooling

### Backup & Recovery

*   [ ]  Back up DuckDB cache database
*   [ ]  Back up configuration files
*   [ ]  Document recovery procedures
*   [ ]  Test restore process
*   [ ]  Set up automated backups

### High Availability

*   [ ]  Run multiple replicas (3+ for production)
*   [ ]  Use load balancer with health checks
*   [ ]  Configure pod/service disruption budgets
*   [ ]  Set up multi-region if needed
*   [ ]  Test failover scenarios

## Troubleshooting

### Port Already in Use

```
# Find process using port 8080lsof -i :8080sudo netstat -tulpn | grep 8080# Kill process or use different portflapi -c config.yaml --port 8081
```

### Permission Denied (Docker)

```
# Run with correct user permissionsdocker run --user $(id -u):$(id -g) \  -v $(pwd)/config:/config \  ghcr.io/datazoode/flapi:latest
```

### Cache Not Persisting

```
# Ensure volume mount for DuckDB-v $(pwd)/data:/data# Check DuckDB path in configduckdb:  db_path: /data/flapi.db  # Must be on mounted volume
```

### Memory Issues

```
# Increase container memory limitsdocker run -m 4g ghcr.io/datazoode/flapi:latest# Or in Kubernetesresources:  limits:    memory: "4Gi"
```

### Configuration Not Found

**Error**: `flapi.yaml not found` or `Config file missing`

```
# Check if config is mounteddocker exec CONTAINER_ID ls -la /config# Should show:# /config/flapi.yaml# /config/sqls/# Fix: Ensure volume mount is correctdocker run -v $(pwd)/config:/config:ro ...# Or check ConfigMap in Kuberneteskubectl describe configmap flapi-config -n flapi
```

### SQL Templates Not Loading

**Error**: `No endpoints found` or `Template directory empty`

```
# Verify template path in flapi.yamltemplate:  path: '/config/sqls'  # Must match mounted path# Check files existdocker exec CONTAINER_ID ls -la /config/sqls# Ensure YAML files have correct structurecat config/sqls/users.yaml# Must contain:# url-path: /users# method: GET# template-source: users.sql# connection:#   - <connection-name from flapi.yaml>## The SQL itself lives in a separate file (e.g. config/sqls/users.sql).
```

### Invalid YAML Syntax

**Error**: `YAML parse error` or `Invalid configuration`

```
# Validate YAML syntax locallyyamllint config/flapi.yaml# Common issues:# - Incorrect indentation (use spaces, not tabs)# - Missing quotes around special characters# - Trailing whitespace# Test configuration before deployingdocker run --rm \  -v $(pwd)/config:/config \  ghcr.io/datazoode/flapi:latest \  -c /config/flapi.yaml --validate-config
```

### Connection Refused / Database Unreachable

**Error**: `Connection refused` or `Host unreachable`

```
# Inside Docker, host.docker.internal points to the host machine.# Connections are configured with init: (SQL to load the extension & ATTACH)# and properties: (key/value pairs surfaced to templates as {{ conn.<key> }}).connections:  postgres:    init: |      INSTALL postgres;      LOAD postgres;      ATTACH 'host=host.docker.internal port=5432 dbname=mydb user=app password=${DB_PASSWORD}'        AS pgdb (TYPE postgres);# In Kubernetes, point at the service DNS name instead:connections:  postgres:    init: |      INSTALL postgres;      LOAD postgres;      ATTACH 'host=postgres.default.svc.cluster.local port=5432 dbname=mydb user=app password=${DB_PASSWORD}'        AS pgdb (TYPE postgres);
```

```
# Test connectivity from inside the containerdocker exec CONTAINER_ID ping postgres-host
```

### ConfigMap Size Limit Exceeded

**Error**: `ConfigMap too large` (> 1MB)

```
# Solution 1: Split into multiple ConfigMapskubectl create configmap flapi-sqls-users --from-file=./config/sqls/users/kubectl create configmap flapi-sqls-products --from-file=./config/sqls/products/# Solution 2: Use PersistentVolume insteadapiVersion: v1kind: PersistentVolumeClaimmetadata:  name: flapi-configspec:  accessModes:    - ReadOnlyMany  resources:    requests:      storage: 1Gi# Upload config to PV manually or via init container
```

### Read-Only Filesystem Errors

**Error**: `Read-only file system` when writing cache

```
# Ensure data directory is writabledocker run -v $(pwd)/data:/data ...  # No :ro flag# In Kubernetes, use ReadWriteOnce PVC for /datavolumes:- name: data  persistentVolumeClaim:    claimName: flapi-data  # Must be RWO, not RO
```

### Environment Variable Not Working

**Error**: Config value not overridden by env var

```
# Check environment variables are setdocker exec CONTAINER_ID env | grep FLAPI# Verify variable names match flAPI conventions# Use UPPERCASE with underscoresFLAPI_DB_PATH=/data/custom.db  # ✅ Correctflapi-db-path=/data/custom.db  # ❌ Wrong# In Kubernetes, check pod envkubectl describe pod flapi-xxx -n flapi | grep -A5 Environment
```

### Cloud Storage Mount Failing

**Google Cloud Run**:

```
# Ensure service account has Storage Object Viewer rolegcloud projects add-iam-policy-binding PROJECT_ID \  --member=serviceAccount:SERVICE_ACCOUNT \  --role=roles/storage.objectViewer# Check bucket permissionsgsutil iam get gs://YOUR_BUCKET# Verify files are in bucketgsutil ls gs://YOUR_BUCKET/flapi-config/
```

**AWS Lambda (EFS)**:

```
# Check EFS mount target is in same VPC as Lambdaaws efs describe-mount-targets --file-system-id fs-xxx# Verify Lambda has VPC permissions# Security groups must allow NFS (port 2049)# Check EFS access pointaws efs describe-access-points --file-system-id fs-xxx
```

### Startup Crashes / Immediate Exit

**Error**: Container starts then exits immediately

```
# Check logs for actual errordocker logs CONTAINER_IDkubectl logs pod/flapi-xxx -n flapi# Common causes:# 1. Missing config file# 2. Invalid YAML syntax# 3. Database connection failure# 4. Port already in use# Run interactively to debugdocker run -it --rm \  -v $(pwd)/config:/config \  ghcr.io/datazoode/flapi:latest \  sh# Then manually test:/app/flapi -c /config/flapi.yaml
```

## Next Steps

*   **[Configuration](/docs/getting-started/configuration.md)**: Learn about flapi.yaml options
*   **[Endpoints](/docs/endpoints/overview.md)**: Create your first API endpoints
*   **[Caching](/docs/guides/caching/setup.md)**: Optimize performance with caching
*   **[Authentication](/docs/endpoints/authentication.md)**: Secure your APIs

Need help with deployment? Check out our [professional services](/services) or [join the community](https://github.com/datazoode/flapi/discussions).

([🍪 Cookie Settings](#cookie-settings))

# Creating Your First API

This guide walks you through creating your first API endpoint with flAPI using a Parquet file as the data source. You will end up with a running `GET /customers/` endpoint that supports optional filtering parameters.

## Prerequisites

Before starting, make sure you have:

1.  The `flapi` binary or Docker image available (see the [Quickstart Guide](/docs/getting-started/quickstart.md)).
2.  A `customers.parquet` file. The flAPI repository ships one under `examples/data/customers.parquet` (TPC-H customer columns: `c_custkey`, `c_name`, `c_acctbal`, `c_mktsegment`, etc.).

## Project Layout

Create the following directory layout:

```
my-first-api/├── flapi.yaml├── data/│   └── customers.parquet└── sqls/    ├── customers.yaml      # Endpoint configuration    └── customers.sql       # SQL template (Mustache)
```

## Step 1: Create the Main Configuration

Create `flapi.yaml` at the project root:

```
project-name: my-first-apiproject-description: My first flAPI projecttemplate:  path: ./sqls  environment-whitelist:    - '^FLAPI_.*'connections:  customers-parquet:    properties:      path: ./data/customers.parquetduckdb:  access_mode: READ_WRITE
```

Key points:

*   `project-name` / `project-description` use hyphens (this is the canonical naming convention).
*   `connections.<name>.properties` exposes values to your SQL template as `{{ conn.<key> }}`.
*   DuckDB pass-through keys (`access_mode`, `db_path`, `threads`, `max_memory`, `default_order`) keep `snake_case` because they are forwarded verbatim to DuckDB.

## Step 2: Define the Endpoint

Create `sqls/customers.yaml`:

```
url-path: /customers/method: GETrequest:  - field-name: id    field-in: query    description: Customer key (c_custkey)    required: false    validators:      - type: int        min: 1        max: 1000000        preventSqlInjection: true  - field-name: segment    field-in: query    description: Market segment    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY]template-source: customers.sqlconnection:  - customers-parquetwith-pagination: true
```

Key points:

*   `url-path`, `template-source`, and `connection` are required.
*   `method` defaults to `GET` if omitted.
*   Each request field declares `field-name`, `field-in` (`query`, `path`, `header`, or `body`), and an optional list of `validators`. Built-in validator types are `int`, `string`, `enum`, `email`, `uuid`, `date`, and `time`.
*   `with-pagination: true` automatically accepts `limit` and `offset` query parameters.

## Step 3: Write the SQL Template

Create `sqls/customers.sql`. Templates use **Mustache** syntax — `{{ params.foo }}` for variables and `{{#params.foo}} … {{/params.foo}}` for conditional sections that render only when the parameter is present.

```
SELECT  c_custkey   AS id,  c_name      AS name,  c_acctbal   AS balance,  c_mktsegment AS segmentFROM '{{{ conn.path }}}'WHERE 1=1{{#params.id}}  AND c_custkey = {{{ params.id }}}{{/params.id}}{{#params.segment}}  AND c_mktsegment = '{{{ params.segment }}}'{{/params.segment}}
```

Notes on the template syntax:

*   `{{{ ... }}}` (triple braces) emits the value **unescaped**, which is required for SQL fragments and file paths.
*   `{{ conn.path }}` reads the `path` property of the `customers-parquet` connection defined in `flapi.yaml`.
*   `{{#params.id}} … {{/params.id}}` renders the inner SQL only when the `id` query parameter is provided. Otherwise it is silently skipped.
*   Validators defined in `customers.yaml` run **before** the template is rendered, so by the time `{{ params.id }}` reaches the SQL it has already been type-checked.

## Step 4: Run flAPI

Start the server pointing at your config file:

```
./flapi -c flapi.yaml
```

flAPI listens on port `8080` by default (override with `-p <port>` or `http-port:` in `flapi.yaml`). You can also validate the configuration without starting the server:

```
./flapi -c flapi.yaml --validate-config
```

## Step 5: Call the Endpoint

In another terminal:

```
# All customers (first page)curl "http://localhost:8080/customers/"# Filter by customer keycurl "http://localhost:8080/customers/?id=42"# Filter by market segmentcurl "http://localhost:8080/customers/?segment=AUTOMOBILE"# Combine filters with paginationcurl "http://localhost:8080/customers/?segment=BUILDING&limit=10&offset=0"
```

## Next Steps

*   Learn about [global configuration options](/docs/getting-started/configuration.md), including DuckLake caching, MCP, and authentication.
*   Explore the [Quickstart Guide](/docs/getting-started/quickstart.md) for a condensed end-to-end walk-through.
*   Read the upstream [Configuration Reference](https://github.com/datazoode/flapi/blob/main/docs/CONFIG_REFERENCE.md) and [CLI Reference](https://github.com/datazoode/flapi/blob/main/docs/CLI_REFERENCE.md) for the complete option list.

([🍪 Cookie Settings](#cookie-settings))

# Introduction to flAPI

**flAPI** is a high-performance serving layer that transforms your enterprise data systems—like Snowflake, BigQuery, or SAP—into fast, secure, AI-ready APIs. It acts as powerful middleware that decouples slow, expensive backend systems from the high-throughput demands of modern applications and AI agents.

## The Problem: Data Warehouses Aren't API Servers

Traditional data warehouses and ERPs excel at analytical workloads but struggle with the demands of modern applications:

*   **High Latency**: Queries take seconds, not milliseconds
*   **Expensive**: Every query costs money; high-frequency access gets expensive fast
*   **Not Designed for APIs**: No built-in rate limiting, caching, or authentication
*   **Poor Concurrency**: Can't handle hundreds of simultaneous API requests
*   **AI-Unfriendly**: No structured tool interfaces for AI agents

Applications and AI agents need fast, frequent access to data. Querying warehouses directly is both slow and expensive.

## The Solution: A High-Performance Serving Layer

flAPI solves this by sitting between your slow backends and fast consumers:

### How It Works

1.  **Connect**: Point flAPI to your data sources (BigQuery, Snowflake, Parquet, etc.)
2.  **Define**: Write SQL templates and configure endpoints in simple YAML files
3.  **Serve**: flAPI materializes a fast, local cache using DuckDB and serves APIs at millisecond latency

The cache absorbs high-frequency queries, dramatically reducing warehouse costs while providing the speed modern applications demand.

## Key Benefits

### 🚀 **Performance**

Millisecond API responses vs. seconds for direct warehouse queries. Your applications feel instant.

### 💰 **Cost Reduction**

Reduce warehouse query costs by 90%+ by serving from cache. One expensive query refreshes data for thousands of API calls.

### 🤖 **AI-Native**

Built-in Model Context Protocol (MCP) support. Expose SQL templates as structured tools for AI agents—no duplicate work.

### 🛠️ **Developer Experience**

Local-first development with powerful CLI (`flapii`) and VS Code integration. Test templates, validate configs, debug queries—all locally.

### 🔒 **Enterprise Security**

JWT authentication, row-level security, rate limiting, and fine-grained access control out of the box.

### 📦 **Simple Deployment**

Single binary, zero dependencies. Deploy anywhere: AWS Lambda, Cloud Run, Kubernetes, or on-premise.

## Core Features

### Declarative API Configuration

Define endpoints, parameters, and security rules in simple YAML files. No code required.

### Full CRUD Support

Create, read, update, and delete endpoints with built-in validation, transactions, and RETURNING clauses. Write data safely with pre-write validation and ACID guarantees.

### SQL as a Transformation Layer

Use the full power of SQL to transform and shape data. flAPI makes it easy to perform significant transformations right in the template.

### Dynamic Templates with Mustache

Create flexible, reusable SQL queries with Mustache syntax—familiar to anyone who's used dbt.

### DuckDB-Powered Caching

Strategic caching that's not just for speed—it's for cost optimization. Materialize high-value data locally and refresh on your schedule. Write operations update both cache and backend seamlessly.

### MCP for AI Agents

Automatically expose your SQL templates as tools that AI agents can call. One config creates both REST APIs and structured tools. Agents can now not just read data but also create, update, and delete records.

### Automatic Documentation

OpenAPI (Swagger) documentation generated automatically from your endpoint configs.

## Who Is flAPI For?

### Data Engineers

Build data APIs in minutes, not weeks. Focus on data transformation, not API boilerplate.

### Backend Developers

Decouple applications from slow data sources. Get the speed you need without rewriting queries.

### AI/ML Teams

Give your agents structured access to enterprise data through MCP. No custom integrations required.

### Platform Teams

Provide a standardized way for teams to expose data as APIs with built-in governance and security.

## Real-World Use Cases

*   **Analytics APIs**: Serve dashboards and reports without hammering your warehouse
*   **Operational APIs**: Create order, customer, and inventory endpoints that update data directly in your warehouse
*   **AI Agent Tools**: Let agents query enterprise data through structured interfaces AND modify records safely with validation
*   **Customer-Facing APIs**: Provide fast data access in customer applications with form submissions and real-time updates
*   **Internal Tools**: Power internal apps with fresh data without warehouse costs, with full read/write access
*   **Real-Time Reporting**: Cache frequently-accessed reports and refresh on schedule
*   **Bidirectional Sync**: Mobile apps and web clients update the cache, which syncs safely to your warehouse

## Architecture Philosophy

flAPI embraces the "small data" philosophy: most applications don't need all your data, just the right slice of it. By materializing a fast, local cache of high-value data, flAPI makes "big data" feel small and fast.

### Local-First Development

Modern hardware is powerful. Develop locally with the exact same software you ship to production. No cloud complexity during development.

### Think Small, Ship Joyfully

The entire workflow is optimized for rapid iteration. Define your API, run `flapi`, and test it instantly. This tight feedback loop makes development a joy.

### Optimized for Serverless

The same small footprint and millisecond startup time make flAPI perfect for cost-efficient serverless deployments.

## Next Steps

Ready to get started?

1.  **[Quickstart Guide](/docs/getting-started/quickstart.md)**: Build your first API in 5 minutes
2.  **[How It Works](/docs/concepts/how-it-works.md)**: Understand the 3-step process
3.  **[Build a Complete CRUD API](/docs/examples/crud-api.md)**: Create, read, update, and delete endpoints
4.  **[Write Operations Reference](/docs/endpoints/write-operations.md)**: Validation, transactions, and RETURNING clauses
5.  **[Architecture Deep Dive](/docs/concepts/architecture.md)**: Technical details and performance
6.  **[Configuration Guide](/docs/getting-started/configuration.md)**: Learn all configuration options
7.  **[Connect Data Sources](/docs/guides/connections/overview.md)**: BigQuery, PostgreSQL, Parquet, SAP
8.  **[Deployment Guide](/docs/getting-started/deployment.md)**: Deploy to production

Need help? Check out our [GitHub Discussions](https://github.com/datazoode/flapi/discussions) or view our [services page](/services) for enterprise support options.

([🍪 Cookie Settings](#cookie-settings))

# Quickstart

## What is Flapi?

Flapi is a lightweight framework that helps you create REST APIs from SQL queries without writing any code. It's particularly useful when you need to serve data from files or databases to applications through a standardized API interface.

## Quick Start Example

Let's solve a common problem: You have customer data in a Parquet file and want to serve it through a REST API with proper validation and filtering capabilities.

### 1\. Get Flapi

You can either download the binary:

```
curl -L https://github.com/datazoode/flapi/releases/latest/download/flapi -o flapichmod +x flapi
```

Or use Docker:

```
docker pull ghcr.io/datazoode/flapi:latest
```

### 2\. Create Minimal Configuration

Create a `flapi.yaml` file:

```
project-name: customer-apiproject-description: API for customer datatemplate:  path: './sqls'connections:  customers-parquet:    properties:      path: './data/customers.parquet'duckdb:  access_mode: READ_WRITE
```

### 3\. Create Your First Endpoint

1.  Create the SQL templates directory:

```
mkdir sqls
```

2.  Create an endpoint configuration (`sqls/customers.yaml`):

```
url-path: /customers/request:  - field-name: id    field-in: query    description: Customer ID    required: false    validators:      - type: int        min: 1template-source: customers.sqlconnection:   - customers-parquet
```

3.  Create the SQL template (`sqls/customers.sql`). flAPI uses **Mustache** templating — `{{ params.foo }}` for variables, `{{#params.foo}} … {{/params.foo}}` for conditional sections, and triple-brace `{{{ ... }}}` for unescaped output (used for SQL fragments and paths):

```
SELECT *FROM '{{{ conn.path }}}'WHERE 1=1{{#params.id}}  AND c_custkey = {{{ params.id }}}{{/params.id}}
```

### 4\. Run Flapi

Using the binary:

```
./flapi -c flapi.yaml
```

Or with Docker:

```
docker run -it --rm -p 8080:8080 -v $(pwd):/config \  ghcr.io/datazoode/flapi -c /config/flapi.yaml
```

Your API is now available at `http://localhost:8080/customers/`!

## Why Flapi?

Data teams often face challenges when sharing data with applications and services. Traditional approaches require:

1.  Writing custom API code
2.  Implementing data validation
3.  Setting up authentication
4.  Managing database connections
5.  Handling error cases
6.  Writing documentation

Flapi solves these challenges by providing:

### Declarative API Definition

*   Define APIs using YAML configuration
*   Automatic parameter validation
*   Built-in SQL injection prevention
*   OpenAPI documentation generation

### Powerful Data Access

*   Connect to Parquet files, DuckDB, and BigQuery
*   SQL templating with Mustache
*   Query result caching
*   Connection pooling

### Enterprise Features

*   Authentication (Basic Auth)
*   Rate limiting
*   CORS support
*   HTTPS enforcement
*   Health checks

## Main Features

### Data Sources

*   **File Formats**: Direct connection to Parquet files
*   **Databases**: DuckDB and BigQuery support
*   **Extensible**: Plugin system for additional data sources

### API Features

*   **Parameter Validation**: Type checking, ranges, enums, regex
*   **SQL Templates**: Mustache templating for dynamic queries
*   **Caching**: Query result caching with DuckDB
*   **Authentication**: Basic auth with user roles
*   **Rate Limiting**: Configurable request limits

### Developer Experience

*   **Zero Code**: Create APIs with just YAML and SQL
*   **OpenAPI**: Automatic API documentation
*   **Docker Support**: Easy deployment with containers
*   **Monitoring**: Built-in health checks and metrics

## Next Steps

Now that you have your first API endpoint running, you can:

1.  **[Create Your First API](/docs/getting-started/first-api.md)**: Build a complete API with validation
2.  **[Configuration Guide](/docs/getting-started/configuration.md)**: Learn all configuration options
3.  **[SQL Templating](/docs/concepts/sql-templating.md)**: Master dynamic queries
4.  **[Connect Data Sources](/docs/guides/connections/overview.md)**: Connect to BigQuery, PostgreSQL, and more
5.  **[Authentication](/docs/endpoints/authentication.md)**: Secure your API
6.  **[Caching](/docs/guides/caching/setup.md)**: Improve performance and reduce costs
7.  **[Deployment](/docs/getting-started/deployment.md)**: Deploy to production

([🍪 Cookie Settings](#cookie-settings))

# Bidirectional Data Sync Patterns

Traditional data serving layers are read-only—they pull data from slow backends and serve it from a fast cache, but changes flow only one direction. flAPI's write support enables **bidirectional sync patterns** where data flows both ways: reads from the cache, writes that update both the cache and the backend.

This guide covers patterns for keeping your flAPI cache and backend data warehouse in sync.

## The Problem with One-Way Sync

Traditional data architectures face challenges:

```
Warehouse → Cache → Consumers (one-way flow)            ↓        stale data        lost updates        dual-source-of-truth
```

When consumers can only read from the cache:

*   Updates to the backend don't immediately reflect in the cache
*   Updates to the cache have no way back to the warehouse
*   You need separate systems to sync data back
*   Risk of conflicting data between cache and warehouse

## The Solution: Bidirectional Sync

With flAPI's write support, data can flow both ways:

```
Warehouse ↔ Cache ↔ Consumers(read & write in both directions)
```

Benefits:

*   **Single source of truth** - All reads and writes go through flAPI
*   **Immediate consistency** - Updates reflected instantly
*   **Simplified architecture** - No separate sync pipelines
*   **Transaction safety** - ACID guarantees on all operations
*   **Audit trail** - All changes logged in warehouse

## Pattern 1: Write-Through

**When:** Need immediate consistency between cache and warehouse

**How:** Every write goes directly to the warehouse, then updates the cache

```
Consumer request to write    ↓Validate in flAPI    ↓Write to Warehouse    ↓Update Cache (if separate)    ↓Return result to consumer
```

### Implementation

**Endpoint configuration:**

```
url-path: /api/customers/:customer_idmethod: PUToperation:  type: Write  validate-before-write: true  returns-data: true  transaction: truerequest:  - field-name: customer_id    field-in: path    required: true    validators:      - type: int        min: 1  - field-name: email    field-in: body    required: false    validators:      - type: string        regex: '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'template-source: customer-update.sqlconnection: [snowflake]  # Write directly to warehouse
```

**SQL:**

```
-- Direct write to SnowflakeUPDATE customersSET  email = COALESCE(:email, email),  updated_at = CURRENT_TIMESTAMP()WHERE customer_id = :customer_idRETURNING *
```

**Advantages:**

*   ✅ Immediate consistency
*   ✅ No sync delay
*   ✅ Simple to understand

**Disadvantages:**

*   ❌ Slower than writing to local cache (warehouse latency)
*   ❌ Network failures block the write
*   ❌ Can't write if warehouse is unavailable

## Pattern 2: Write-Back (Eventual Consistency)

**When:** Need fast writes with eventual consistency

**How:** Write to local cache immediately, sync to warehouse asynchronously

```
Consumer writes to flAPI    ↓Write to DuckDB Cache (fast)    ↓Return to consumer immediately    ↓[Async] Sync to Warehouse    ↓[Async] Handle conflicts
```

### Implementation

**Endpoint writes to DuckDB:**

```
url-path: /api/orders/method: POSToperation:  type: Write  validate-before-write: true  returns-data: true  transaction: truerequest:  - field-name: customer_id    required: true    validators:      - type: int        min: 1  - field-name: total_amount    required: true    validators:      - type: string        regex: '^\d+(\.\d{2})?$'template-source: order-create.sqlconnection: [local-duckdb]  # Write to fast local cache
```

**SQL:**

```
INSERT INTO orders (  customer_id,  total_amount,  status,  created_at,  synced_to_warehouse)VALUES (  :customer_id,  :total_amount,  'pending',  CURRENT_TIMESTAMP(),  false)RETURNING *
```

**Separate sync process:**

Create a scheduled job to sync pending writes to the warehouse:

```
-- Sync pending orders to SnowflakeINSERT INTO snowflake.ordersSELECT * FROM orders WHERE synced_to_warehouse = false-- Mark as syncedUPDATE orders SET synced_to_warehouse = trueWHERE synced_to_warehouse = false-- Handle conflicts with MAX(updated_at)
```

**Advantages:**

*   ✅ Ultra-fast writes (local cache only)
*   ✅ Resilient to warehouse outages
*   ✅ Great user experience

**Disadvantages:**

*   ❌ Eventual consistency (temporary divergence)
*   ❌ Requires handling sync failures
*   ❌ Complex conflict resolution

## Pattern 3: Read-Through Write-Back Hybrid

**When:** Need balance between speed and consistency

**How:** Reads from cache, writes batch to warehouse on schedule

```
Consumer reads    ↓Read from Cache (fast)    ↓Consumers write    ↓Write to Cache (fast)    ↓[Scheduled] Batch sync to Warehouse    ↓Update cache on refresh
```

### Implementation

**Read endpoints (from cache):**

```
url-path: /api/products/method: GETrequest: []template-source: products-read.sqlconnection: [local-duckdb]  # Read from fast cache
```

**Write endpoints (to cache with sync flag):**

```
url-path: /api/products/:product_idmethod: PUToperation:  type: Write  validate-before-write: true  returns-data: true  transaction: truerequest:  - field-name: product_id    field-in: path    required: true  - field-name: unit_price    field-in: body    required: falsetemplate-source: product-update.sqlconnection: [local-duckdb]
```

**SQL with sync tracking:**

```
UPDATE productsSET  unit_price = COALESCE(:unit_price, unit_price),  needs_sync = true,        -- Flag for batch sync  updated_at = NOW()WHERE product_id = :product_idRETURNING *
```

**Scheduled batch sync job (every 5 minutes):**

```
BEGIN TRANSACTION;-- Copy changes to warehouseINSERT INTO warehouse.productsSELECT * FROM products WHERE needs_sync = trueON CONFLICT (product_id) DO UPDATE SET  unit_price = EXCLUDED.unit_price,  updated_at = EXCLUDED.updated_at;-- Mark as syncedUPDATE products SET needs_sync = falseWHERE needs_sync = true;COMMIT;
```

**Advantages:**

*   ✅ Fast reads from cache
*   ✅ Fast writes to cache
*   ✅ Batch efficiency
*   ✅ Reasonable consistency window

**Disadvantages:**

*   ❌ Sync delay (usually 5-15 minutes)
*   ❌ Requires batch reconciliation logic
*   ❌ Storage overhead for sync tracking

## Pattern 4: Smart Cache Invalidation

**When:** Need to keep cache fresh after writes

**How:** Invalidate relevant cache entries, lazy-reload on next read

```
Write to Cache    ↓Invalidate Cache Entry    ↓Next Read    ↓Cache Miss    ↓Reload from Warehouse    ↓Serve to Consumer
```

### Implementation

**Write endpoint that invalidates and optionally refreshes the cache:**

```
url-path: /api/customers/:customer_idmethod: PUToperation:  type: Write  validate-before-write: true  returns-data: true  transaction: true# After successful write, invalidate / refresh the endpoint's DuckLake cachecache:  invalidate-on-write: true     # mark cache as stale  refresh-on-write: false       # set true to force a refresh nowtemplate-source: customer-update.sqlconnection: [snowflake]
```

**Cache-aware read endpoint (DuckLake):**

```
url-path: /api/customers/:customer_idmethod: GETcache:  enabled: true  table: customers_cache  schedule: 1h  primary-key: [customer_id]  cursor:    column: updated_at    type: timestamptemplate-source: customer-read.sqlconnection: [snowflake]
```

**Advantages:**

*   ✅ Automatic cache freshness
*   ✅ Lazy reloading (only on demand)
*   ✅ Reduced memory footprint

**Disadvantages:**

*   ❌ First read after write is slow
*   ❌ Requires cache layer (Redis, Memcached)
*   ❌ More complex setup

## Conflict Resolution Strategies

When syncing writes from cache back to warehouse, conflicts can occur.

### Strategy 1: Last-Write-Wins

Simplest approach: later timestamp wins.

```
INSERT INTO warehouse.dataSELECT * FROM cache.data WHERE needs_sync = trueON CONFLICT (id) DO UPDATE SET  -- Take value from cache if warehouse update is older  value = CASE    WHEN warehouse.updated_at < EXCLUDED.updated_at    THEN EXCLUDED.value    ELSE warehouse.value  END,  updated_at = GREATEST(    warehouse.updated_at,    EXCLUDED.updated_at  );
```

### Strategy 2: Custom Merge Logic

Different rules for different columns.

```
-- For price: take max (don't lower prices accidentally)-- For availability: take min (conservative estimate)-- For notes: concatenate with timestampINSERT INTO warehouse.productsSELECT * FROM cache.products WHERE needs_sync = trueON CONFLICT (product_id) DO UPDATE SET  unit_price = GREATEST(warehouse.unit_price, EXCLUDED.unit_price),  units_in_stock = LEAST(warehouse.units_in_stock, EXCLUDED.units_in_stock),  notes = warehouse.notes || ' | ' || EXCLUDED.notes,  updated_at = CURRENT_TIMESTAMP();
```

### Strategy 3: Manual Conflict Resolution

Flag conflicts for human review.

```
-- Detect conflictsWITH conflicting_updates AS (  SELECT    cache.id,    cache.value as cache_value,    warehouse.value as warehouse_value  FROM cache.data cache  INNER JOIN warehouse.data warehouse    ON cache.id = warehouse.id    AND cache.value != warehouse.value    AND cache.updated_at > warehouse.updated_at)INSERT INTO conflict_log (record_id, cache_value, warehouse_value, detected_at)SELECT id, cache_value, warehouse_value, NOW()FROM conflicting_updates;
```

## Best Practices

### 1\. Choose Your Consistency Model

Decide: **Immediate** (write-through) vs **Eventual** (write-back)?

*   High-stakes data (payments, inventory) → write-through
*   User-generated content (profiles, preferences) → write-back
*   Reporting data → write-back with batch sync

### 2\. Always Validate Before Writing

Prevent invalid data entering either cache or warehouse:

```
operation:  validate-before-write: true
```

### 3\. Use Transactions

Ensure partial operations don't corrupt data:

```
operation:  transaction: true
```

### 4\. Track Sync Status

Make it easy to identify what's synced:

```
CREATE TABLE products (  id INTEGER,  name STRING,  updated_at TIMESTAMP,  synced_at TIMESTAMP,        -- NULL until synced  needs_sync BOOLEAN);
```

### 5\. Monitor Divergence

Alert when cache and warehouse drift too far:

```
SELECT COUNT(*) as diverged_recordsFROM cache.products cFULL OUTER JOIN warehouse.products w  ON c.id = w.idWHERE c.updated_at != w.updated_at  OR c.needs_sync = true;
```

### 6\. Set Reasonable Refresh Schedules and Retention

DuckLake doesn't use a TTL — it uses snapshot-based refresh plus retention:

```
cache:  enabled: true  schedule: 1h            # Refresh hourly  retention:    keep-last-snapshots: 24    max-snapshot-age: 24h
```

## Examples

### Example 1: User Profile Sync

Fast writes to cache, sync to warehouse every 10 minutes:

```
# Endpoint: PUT /api/users/:user_idoperation:  type: Write  returns-data: true  transaction: true# Writes to local DuckDB for speedconnection: [local-duckdb]
```

Sync to Snowflake is performed by an external scheduled job (cron, Airflow, etc.) that reads the local table and merges rows where `needs_sync = true` into the warehouse — flAPI itself does not provide a built-in cross-connection sync scheduler.

### Example 2: Order Processing

Immediate consistency: write directly to warehouse:

```
# Endpoint: POST /api/orders/operation:  type: Write  returns-data: true  transaction: true# Write directly to Snowflakeconnection: [snowflake]# No sync delay needed
```

### Example 3: Analytics Data

Batch updates with smart cache invalidation:

```
# Endpoint: PUT /api/metrics/:metric_idoperation:  type: Write  returns-data: true  transaction: true# Write to cacheconnection: [local-duckdb]# Invalidate this endpoint's DuckLake cache after the writecache:  invalidate-on-write: true  refresh-on-write: false
```

A separate scheduled job batches the warehouse sync; flAPI exposes the audit table `<catalog>.audit.sync_events` to monitor refreshes.

## When to Use Each Pattern

| Pattern | Speed | Consistency | Complexity | Use Case |
| --- | --- | --- | --- | --- |
| Write-Through | Slow | Immediate | Low | Critical data (payments) |
| Write-Back | Very Fast | Eventual | High | User data, preferences |
| Hybrid | Fast | Delayed | Medium | Analytics, reporting |
| Cache Invalidation | Fast | Hours | Medium | Public data, products |

## Related Topics

*   [Write Operations Reference](/docs/endpoints/write-operations.md)
*   [Complete CRUD API Tutorial](/docs/examples/crud-api.md)
*   [Caching Strategy](/docs/concepts/caching-strategy.md)

([🍪 Cookie Settings](#cookie-settings))

# Cloud Storage and VFS

flAPI can load its main configuration file (`flapi.yaml`), endpoint definitions, and SQL templates directly from cloud object storage or any HTTPS URL. Under the hood it uses DuckDB's Virtual File System (VFS), so anything DuckDB can read, flAPI can read.

## Why

Treating configuration as a remote artifact unlocks a few useful patterns:

*   **Serverless and read-only containers** — run flAPI on AWS Lambda, Cloud Run, or Container Apps with no mounted volumes; just point `--config` at a bucket.
*   **GitOps and CI-driven config rollouts** — push a YAML change to S3 and roll endpoints without rebuilding the container image.
*   **Multi-environment fan-out** — one image, three buckets: `s3://configs-dev/`, `s3://configs-staging/`, `s3://configs-prod/`.
*   **Centralized template library** — share `sqls/` templates across many flAPI deployments.

## Quick Start

One example per supported scheme. Set credentials in the environment first (see ([Authentication](#authentication)) below), then:

```
# Local file (default)flapi --config ./flapi.yaml# HTTPS (no credentials required for public URLs)flapi --config https://raw.githubusercontent.com/myorg/configs/main/flapi.yaml# Amazon S3flapi --config s3://my-bucket/configs/flapi.yaml# Google Cloud Storageflapi --config gs://my-bucket/configs/flapi.yaml# Azure Blob Storageflapi --config az://my-container/configs/flapi.yaml
```

## Supported URI Schemes

| Scheme | Description | Credential source |
| --- | --- | --- |
| `file://` | Local filesystem (also any plain path) | Filesystem permissions |
| `https://` | HTTPS URL (public or with auth headers) | None required |
| `http://` | HTTP URL — not recommended for production | None required |
| `s3://` | Amazon S3 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` |
| `gs://` | Google Cloud Storage | `GOOGLE_APPLICATION_CREDENTIALS` |
| `az://`, `azure://`, `abfs://`, `abfss://` | Azure Blob Storage | `AZURE_STORAGE_ACCOUNT` + `AZURE_STORAGE_KEY` (or connection string) |

Only `file://` and `https://` are allowed by default. To use any other scheme, declare a connection that installs and configures `httpfs` (see ([Path security](#path-security))).

## Authentication

### Amazon S3

```
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLEexport AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYexport AWS_REGION=us-east-1# Optional: temporary credentials from STS / assumed roleexport AWS_SESSION_TOKEN=FwoGZXIvYXdzE...flapi --config s3://my-bucket/configs/flapi.yaml
```

When flAPI runs on EC2, ECS, EKS, or Lambda it transparently uses the attached IAM role — no environment variables required. For S3-compatible storage (MinIO, LocalStack), set `AWS_ENDPOINT_URL=http://localhost:9000`.

### Google Cloud Storage

```
# Option 1: service-account key fileexport GOOGLE_APPLICATION_CREDENTIALS=/secrets/sa.json# Option 2: application default credentials from gcloudgcloud auth application-default loginflapi --config gs://my-bucket/configs/flapi.yaml
```

On Compute Engine, Cloud Run, or GKE the attached service account is used automatically.

### Azure Blob Storage

```
# Option A: account name + keyexport AZURE_STORAGE_ACCOUNT=mystorageaccountexport AZURE_STORAGE_KEY=base64key==# Option B: full connection stringexport AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=...;EndpointSuffix=core.windows.net"flapi --config az://my-container/configs/flapi.yaml
```

Managed Identity is used automatically on Azure-hosted compute.

## Remote Template Paths

Two ways to load SQL templates from cloud storage.

### 1\. Set `template.path` at the project level

Every endpoint's `template-source` is resolved relative to this base URL.

```
# flapi.yamlproject-name: cloud-templates-demotemplate:  path: s3://my-bucket/templates/
```

```
# endpoint YAML, hosted at s3://my-bucket/templates/customers.yamlurl-path: /customersmethod: GETtemplate-source: customers.sql   # resolves to s3://my-bucket/templates/customers.sqlconnection:  - cloud-data
```

### 2\. Per-endpoint full URI

A `template-source` containing a full URI overrides `template.path`:

```
url-path: /ordersmethod: GETtemplate-source: https://templates.example.com/orders.sqlconnection:  - local-data
```

You can mix and match: local `flapi.yaml`, remote templates; or remote `flapi.yaml`, local templates.

## Path Security

flAPI hardens remote loading with three layers of defense:

1.  **Path traversal blocking** — any path containing `..` is rejected before resolution.
2.  **URL-encoded traversal detection** — encoded variants such as `%2e%2e` are decoded and re-checked.
3.  **Scheme whitelist** — only schemes that are explicitly allowed will be opened.

The defaults are deliberately strict:

| Scheme | Allowed by default? |
| --- | --- |
| `file://` (and plain paths) | yes |
| `https://` | yes |
| `http://`, `s3://`, `gs://`, `az://`, `azure://`, `abfs://`, `abfss://` | **no** — must be enabled |

To enable a cloud scheme, declare a `connection` whose `init:` block installs and loads `httpfs` (DuckDB's cloud filesystem extension). flAPI detects the scheme registration and adds it to the allowlist:

```
connections:  cloud-data:    init: |      INSTALL httpfs;      LOAD httpfs;      SET s3_region='us-east-1';    properties:      bucket: my-data-bucket      base_path: s3://my-data-bucket/data/
```

Once a scheme is allowed by any connection, the same scheme can also be used by `--config` and `template.path`.

## Caching of Remote Files

Remote files are cached locally with an LRU cache so that flAPI doesn't hammer S3/GCS/Azure on every reload. Local files are never cached — they are always read fresh.

Configure caching in the top-level `storage:` block of `flapi.yaml`:

```
storage:  cache:    enabled: true     # default: true    ttl: 300          # cache TTL in seconds, default: 300    max_size: 50MB    # LRU eviction threshold, default: 50MB  credentials:    s3:      type: environment    # environment | secret | instance_profile      region: us-east-1    gcs:      type: environment    # environment | service_account      key_file: /secrets/gcs.json    azure:      type: environment    # environment | connection_string | managed_identity      account: mystorageaccount
```

Caches are invalidated automatically when TTL expires or when the server restarts. They can also be cleared via the config service health endpoint.

## Health Check

The Config Service exposes a health endpoint that reports VFS state — useful as a Kubernetes readiness probe or for incident triage.

```
curl http://127.0.0.1:8080/api/v1/_config/health
```

```
{  "status": "healthy",  "storage": {    "status": "healthy",    "backends": [      {        "name": "config",        "path": "s3://my-bucket/configs/flapi.yaml",        "accessible": true,        "latency_ms": 45,        "scheme": "s3"      },      {        "name": "templates",        "path": "./sqls/",        "accessible": true,        "latency_ms": 2,        "scheme": "local"      }    ],    "total_latency_ms": 47  },  "credentials": {    "s3_configured": true,    "gcs_configured": false,    "azure_configured": false  }}
```

This endpoint does not require authentication, even when the Config Service token is set.

## Complete Worked Example

A fully cloud-native deployment with `flapi.yaml` in S3, endpoint YAML alongside it, and templates in the same bucket.

### `s3://my-bucket/configs/flapi.yaml`

```
project-name: Cloud-Native APIproject-description: flAPI configuration served from S3template:  path: s3://my-bucket/templates/  environment-whitelist:    - '^AWS_.*'storage:  cache:    enabled: true    ttl: 300    max_size: 50MBconnections:  cloud-data:    init: |      INSTALL httpfs;      LOAD httpfs;      SET s3_region='us-east-1';    properties:      base_path: s3://my-data-bucket/parquet/duckdb:  access_mode: READ_ONLY
```

### `s3://my-bucket/templates/customers.yaml`

```
url-path: /customersmethod: GETtemplate-source: customers.sqlconnection:  - cloud-datarequest:  - field-name: country    field-in: query    required: false    validators:      - type: string
```

### `s3://my-bucket/templates/customers.sql`

```
SELECT id, name, email, countryFROM read_parquet('{{ conn.cloud-data.base_path }}/customers.parquet')WHERE 1 = 1{{#params.country}}  AND country = '{{ params.country }}'{{/params.country}}LIMIT 1000
```

### Run

```
export AWS_ACCESS_KEY_ID=...export AWS_SECRET_ACCESS_KEY=...export AWS_REGION=us-east-1flapi --config s3://my-bucket/configs/flapi.yaml
```

flAPI loads `flapi.yaml` from S3, discovers `customers.yaml` under `template.path`, resolves `customers.sql` relative to the same prefix, registers the endpoint, and starts serving `GET /customers?country=DE` — all without any local files.

## Troubleshooting

*   **`Access Denied`** — verify `AWS_ACCESS_KEY_ID` and that the IAM policy grants `s3:GetObject` on the config and templates prefixes.
*   **`File Not Found`** — double-check the bucket/key. `aws s3 ls s3://my-bucket/configs/` is the fastest sanity check.
*   **Slow startup** — use regional endpoints; remote configuration loading adds network latency. The LRU cache absorbs subsequent requests.
*   **Scheme not allowed** — confirm a connection installs and loads `httpfs` so that the scheme makes it onto the allowlist.

## Related

*   [Configuration Service](/docs/tools/configuration-service.md) — file-level YAML reference and validation
*   [Config Service REST API](/docs/tools/config-service-api.md) — runtime endpoint management
*   [flapii CLI](/docs/tools/cli-overview.md) — local validation tooling

([🍪 Cookie Settings](#cookie-settings))

# DuckLake Caching Setup

flAPI's caching layer is built on **DuckLake**, a snapshot-based table format over DuckDB. Each refresh produces a new snapshot, your endpoints serve from the latest snapshot in milliseconds, and a retention policy expires old snapshots automatically.

This guide walks through enabling DuckLake at the project level and configuring per-endpoint caches.

## Step 1: Enable DuckLake Globally

Add a `ducklake:` block to `flapi.yaml`. Without it, endpoint caches do nothing.

```
# flapi.yamlducklake:  enabled: true  alias: cache                         # Catalog alias used in templates  metadata-path: ./data/cache.ducklake # DuckLake metadata directory  data-path: ./data/cache              # DuckLake data files directory  retention:    keep-last-snapshots: 10    max-snapshot-age: 30d  compaction:    enabled: true    schedule: '@daily'  scheduler:    enabled: true    scan-interval: 5m                  # Background scheduler scan interval
```

The `alias` value (`cache`) becomes `{{cache.catalog}}` in every cache template.

## Step 2: Enable Caching on an Endpoint

```
# sqls/customers/customers-rest.yamlurl-path: /customers/cache:  enabled: true  table: customers_cache  schema: analytics            # optional, default "main"  schedule: 5m                 # 30s | 5m | 1h | 2d  primary-key: [id]            # required for merge mode  cursor:    column: registration_date  # required for append/merge    type: date                 # int | date | timestamp  rollback-window: 2d  retention:    keep-last-snapshots: 5    max-snapshot-age: 14d  delete-handling: soft        # soft | hard  template-file: customers_cache.sql   # optional custom refresh SQLrequest:  - field-name: segment    field-in: querytemplate-source: customers.sqlconnection:  - bigquery-warehouse
```

All keys above are verified against `cache_manager.cpp`, `CONFIG_REFERENCE.md` §6 and the `customer-common.yaml` example.

## Step 3: Refresh Modes

flAPI does **not** have a `strategy:` key. The refresh mode is derived from the combination of `cursor` and `primary-key` (`CacheManager::determineCacheMode`):

| `cursor` | `primary-key` | Mode | Behaviour |
| --- | --- | --- | --- |
| absent | _any_ | **full** | Rebuild the cache table on every refresh |
| present | absent | **append** | Append rows past the cursor watermark |
| present | present | **merge** | Reconcile inserts, updates and deletes |

### Full Refresh

```
cache:  enabled: true  table: countries_cache  schedule: 1h
```

### Incremental Append

```
cache:  enabled: true  table: events_cache  schedule: 15m  cursor:    column: created_at    type: timestamp
```

### Incremental Merge

```
cache:  enabled: true  table: customers_cache  schedule: 1m  primary-key: [id]  cursor:    column: updated_at    type: timestamp  delete-handling: soft
```

## Step 4: Write the Cache Template (optional)

If you provide `template-file`, flAPI runs that SQL on each refresh. Otherwise it derives a default. Cache templates have access to a `cache.*` context (verified against `sql_template_processor.cpp`):

| Variable | Description |
| --- | --- |
| `{{cache.catalog}}` | DuckLake catalog alias |
| `{{cache.schema}}` | Cache schema |
| `{{cache.table}}` | Cache table name |
| `{{cache.schedule}}` | Configured refresh schedule |
| `{{cache.mode}}` | `full`, `append`, or `merge` |
| `{{cache.snapshotId}}` | Current snapshot ID |
| `{{cache.snapshotTimestamp}}` | Current snapshot timestamp |
| `{{cache.previousSnapshotId}}` | Previous snapshot ID |
| `{{cache.previousSnapshotTimestamp}}` | Previous snapshot timestamp |
| `{{cache.cursorColumn}}` | `cursor.column` value |
| `{{cache.cursorType}}` | `cursor.type` value |
| `{{cache.primaryKeys}}` | Comma-separated primary key columns |

Real example from `examples/sqls/customers/customers_cache.sql`:

```
-- DuckLake cache templateCREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} ASSELECT    id,    name,    email,    segment,    registration_date,    CURRENT_TIMESTAMP AS cache_updated_at,    '{{cache.snapshotId}}' AS cache_snapshot_idFROM read_parquet('{{conn.path}}')ORDER BY registration_date DESC;
```

Incremental templates can branch on the previous snapshot:

```
INSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT * FROM source_table{{#cache.previousSnapshotTimestamp}}WHERE {{cache.cursorColumn}} > TIMESTAMP '{{cache.previousSnapshotTimestamp}}'{{/cache.previousSnapshotTimestamp}}
```

## Step 5: Write the API Template

The serving endpoint reads from the cache table just like any DuckDB table:

```
-- sqls/customers/customers.sqlSELECT *FROM cache.analytics.customers_cacheWHERE 1=1{{#params.segment}}  AND segment = '{{{params.segment}}}'{{/params.segment}}ORDER BY registration_date DESCLIMIT 100;
```

## Schedule Format

`cache.schedule` accepts a value-unit string parsed by `TimeInterval::parseInterval`:

| Schedule | Use case |
| --- | --- |
| `30s` | Aggressive freshness |
| `5m` | Near real-time dashboards |
| `1h` | Standard BI refresh |
| `6h` | Slow-changing data |
| `24h` | Daily reports |

## Retention and Snapshot Expiry

Two knobs control retention:

```
cache:  retention:    keep-last-snapshots: 10   # keep N newest snapshots    max-snapshot-age: 30d     # also delete anything older
```

When either is set, flAPI calls `ducklake_expire_snapshots` against the catalog after a successful refresh. `rollback-window` reserves snapshots for time-travel queries during the window.

## Audit Trail

`CacheManager::initializeAuditTables` creates `<catalog>.audit.sync_events` at startup. Every refresh records:

*   `endpoint_path`, `cache_table`, `cache_schema`
*   `sync_type` (`full`, `append`, `merge`, `garbage_collection`)
*   `status` (`success`, `error`, `warning`) and `message`
*   `snapshot_id`, `rows_affected`, `sync_started_at`, `sync_completed_at`, `duration_ms`

Query the audit table directly with DuckDB to debug refresh issues.

## Real-World Example

Customer endpoint with DuckLake merge caching (from `examples/sqls/customers/customers-rest.yaml`):

```
url-path: /customers/cache:  enabled: true  table: customers_rest_cache  schema: analytics  schedule: 5m  primary-key: [id]  cursor:    column: registration_date    type: date  retention:    keep-last-snapshots: 3    max-snapshot-age: 7dwith-pagination: true
```

## Best Practices

*   **Start with a `full` refresh**, switch to `append` or `merge` once you have a stable cursor column.
*   **Always set `retention`** — without it, DuckLake snapshot data grows unbounded.
*   **Use `delete-handling: soft`** if downstream consumers need a history of deletions.
*   **Keep `scheduler.scan-interval` small enough** to honour the tightest `cache.schedule` value across endpoints.
*   **Pre-aggregate** in the cache template; don't mirror raw source tables.

## Troubleshooting

### Cache table doesn't exist

*   Confirm `ducklake.enabled: true` and a valid `metadata-path` / `data-path`.
*   Check the startup log for `Initialized DuckLake audit tables` — if absent, DuckLake never came up.
*   Inspect `<catalog>.audit.sync_events` for `status='error'` rows.

### Refresh never fires

*   Confirm `ducklake.scheduler.enabled: true`.
*   `scheduler.scan-interval` must be less than the smallest `cache.schedule` you want to honour.

### Snapshots growing without bound

*   Add `retention.keep-last-snapshots` and/or `retention.max-snapshot-age`.
*   Make sure `compaction.enabled: true` and a `compaction.schedule` is set.

## Next Steps

*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: DuckLake concepts and cost model
*   **[BigQuery Example](/docs/examples/bigquery-caching.md)**: Complete implementation
*   **[SAP ERP Example](/docs/examples/sap-erp-api.md)**: Enterprise caching

([🍪 Cookie Settings](#cookie-settings))

# Connecting to BigQuery

**Extension Credit:**

This guide uses the **[bigquery extension](https://duckdb.org/community_extensions/extensions/bigquery.html)** by the DuckDB community. Thanks to the contributors who made BigQuery integration seamless!

Google BigQuery is one of the most popular cloud data warehouses. flAPI makes it easy to expose BigQuery data as fast, cost-effective APIs through its caching layer.

## Why Use flAPI with BigQuery?

Direct BigQuery access for APIs is problematic:

*   **Slow**: Queries take 2-10 seconds
*   **Expensive**: $0.05+ per query (based on data scanned)
*   **Not designed for high-frequency access**: Poor for serving APIs

**With flAPI caching:**

*   Millisecond responses (1-10ms vs 2-10s)
*   90%+ cost reduction through caching
*   Purpose-built for API serving
*   Query massive datasets efficiently

## DuckDB BigQuery Extension

flAPI uses the [DuckDB BigQuery extension](https://duckdb.org/community_extensions/extensions/bigquery.html) to connect to Google BigQuery. This community extension provides:

*   **Direct BigQuery access** via `bigquery_scan()` function
*   **Google Cloud authentication** (service accounts, ADC)
*   **Push-down optimization** — filters sent to BigQuery
*   **Schema inference** — automatic type mapping
*   **Partition pruning** — efficient data access

## Setup

### 1\. Install BigQuery Extension

```
# flapi.yamlconnections:  bigquery-warehouse:    init: |      INSTALL 'bigquery' FROM community;      LOAD 'bigquery';    properties:      project_id: 'my-project-id'
```

The connection property `project_id` is accessible inside SQL templates as `{{ conn.project_id }}`.

### 2\. Authentication

The DuckDB BigQuery extension uses Google Application Default Credentials (ADC). Set the standard environment variable so the extension picks them up:

```
# Service account (recommended for production)export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json# Or interactive (development)gcloud auth application-default login
```

flAPI does not need to know about the credentials — the extension reads them directly. No `credentials_path:` property is required (or supported) by flAPI's connection block.

## Querying BigQuery

### Query Strategies Comparison

### Direct Query (Without Cache)

```
-- Endpoint templateSELECT  campaign_id,  campaign_name,  SUM(clicks) AS total_clicks,  SUM(revenue) AS total_revenueFROM bigquery_scan('{{ conn.project_id }}.analytics.campaigns')WHERE date >= CURRENT_DATE - INTERVAL 7 DAY{{#params.country}}  AND country = '{{{ params.country }}}'{{/params.country}}GROUP BY 1, 2ORDER BY total_revenue DESC
```

**Cost**: Every API call = 1 BigQuery query ($0.05+)

### With Caching (Recommended)

**Cache Template** (`sqls/campaigns_cache.sql`):

```
-- Runs on schedule (e.g., hourly)CREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} ASSELECT  campaign_id,  campaign_name,  country,  campaign_type,  SUM(clicks) AS total_clicks,  SUM(revenue) AS total_revenue,  MAX(last_updated) AS last_updatedFROM bigquery_scan('{{ conn.project_id }}.analytics.campaigns')WHERE date >= CURRENT_DATE - INTERVAL 30 DAYGROUP BY 1, 2, 3, 4
```

**API Template** (`sqls/campaigns.sql`):

```
SELECT *FROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE 1=1{{#params.country}}  AND country = '{{{ params.country }}}'{{/params.country}}{{#params.campaign_type}}  AND campaign_type = '{{{ params.campaign_type }}}'{{/params.campaign_type}}ORDER BY total_revenue DESCLIMIT 100
```

**Endpoint Configuration** (`sqls/campaigns.yaml`):

```
url-path: /campaigns/method: GETcache:  enabled: true  table: campaigns_cache  schema: analytics  schedule: 60m  template-file: campaigns_cache.sqlrequest:  - field-name: country    field-in: query    required: false    validators:      - type: string        regex: '^[A-Z]{2}$'  - field-name: campaign_type    field-in: query    required: falsetemplate-source: campaigns.sqlconnection:  - bigquery-warehouse
```

## Cost Comparison

### Scenario: Marketing Dashboard API

**Requirements:**

*   10,000 API calls per day
*   BigQuery table: 100GB, costs $0.05 per full scan

**Without flAPI (Direct BigQuery):**

```
10,000 queries × $0.05 = $500/day = $15,000/monthResponse time: 2-5 seconds per query
```

**With flAPI (Hourly Cache):**

```
24 cache refreshes × $0.05 = $1.20/day = $36/monthAPI serving: Free (from cache)Response time: 1-50msSavings: $14,964/month (99.76%)Speed: 1000-10,000x faster
```

## Advanced Patterns

### Incremental Loading

Only load new data to reduce BigQuery costs:

```
-- Cache refresh template (incremental)INSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT *FROM bigquery_scan('{{ conn.project_id }}.dataset.table')WHERE event_timestamp > (  SELECT MAX(event_timestamp)  FROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}})
```

### Partitioned Tables

Query specific partitions to reduce costs:

```
SELECT *FROM bigquery_scan('{{ conn.project_id }}.dataset.events')WHERE _PARTITIONDATE = CURRENT_DATE
```

### Multiple BigQuery Projects

```
connections:  bigquery-prod:    init: |      INSTALL 'bigquery' FROM community;      LOAD 'bigquery';    properties:      project_id: 'prod-project'  bigquery-analytics:    init: |      INSTALL 'bigquery' FROM community;      LOAD 'bigquery';    properties:      project_id: 'analytics-project'
```

## Best Practices

### 1\. Always Use Caching for High-Frequency Access

Direct BigQuery queries are expensive. Use caching unless you need real-time data.

### 2\. Pre-Aggregate in Cache

```
-- Good: Pre-aggregatedCREATE OR REPLACE TABLE {{cache.table}} ASSELECT  date,  country,  SUM(revenue) AS revenueFROM sourceGROUP BY 1, 2-- Bad: Raw data (expensive to cache)CREATE OR REPLACE TABLE {{cache.table}} ASSELECT * FROM source
```

### 3\. Optimize Cache Refresh Schedule

Balance freshness vs cost:

*   **Real-time dashboards**: every 15 minutes
*   **Business reporting**: every hour
*   **Analytics**: daily

### 4\. Use Incremental Updates

For large tables, only refresh changed data:

```
WHERE updated_at >= CURRENT_TIMESTAMP - INTERVAL 24 HOUR
```

## Troubleshooting

### Authentication Errors

```
Error: Failed to authenticate with BigQuery
```

**Solution:**

1.  Check credentials file exists
2.  Verify service account has BigQuery permissions
3.  Ensure `GOOGLE_APPLICATION_CREDENTIALS` is set

```
gcloud auth application-default login
```

### Query Timeout

```
Error: BigQuery query timeout
```

**Solution:**

1.  Optimize your SQL query
2.  Add filters to reduce data scanned
3.  Use partitioned tables

### Cost Alerts

Set up BigQuery cost monitoring in Google Cloud Console:

1.  **Budget Alerts**: get notified when spending exceeds threshold
2.  **Query Quotas**: limit per-user query costs
3.  **flAPI Monitoring**: track cache refresh costs vs API usage

## Example: Complete Setup

**`flapi.yaml`:**

```
project-name: marketing-apitemplate:  path: './sqls'connections:  bigquery-marketing:    init: |      INSTALL 'bigquery' FROM community;      LOAD 'bigquery';    properties:      project_id: 'my-project-id'
```

**`sqls/campaigns.yaml`:**

```
url-path: /campaigns/method: GETcache:  enabled: true  table: campaigns_cache  schema: analytics  schedule: 60m  template-file: campaigns_cache.sqlrequest:  - field-name: country    field-in: query    required: false    validators:      - type: string        regex: '^[A-Z]{2}$'template-source: campaigns.sqlconnection:  - bigquery-marketing
```

**`sqls/campaigns_cache.sql`:**

```
CREATE OR REPLACE TABLE {{cache.catalog}}.{{cache.schema}}.{{cache.table}} ASSELECT  campaign_id,  campaign_type,  country,  SUM(clicks) AS clicks,  SUM(revenue) AS revenueFROM bigquery_scan('{{ conn.project_id }}.marketing.campaigns')WHERE date >= CURRENT_DATE - INTERVAL 30 DAYGROUP BY 1, 2, 3
```

**`sqls/campaigns.sql`:**

```
SELECT *FROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE 1=1{{#params.country}}  AND country = '{{{ params.country }}}'{{/params.country}}ORDER BY revenue DESC
```

**Test it:**

```
# Start flAPI$ ./flapi -c flapi.yaml# Call API$ curl 'http://localhost:8080/campaigns?country=US'
```

## Next Steps

*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: Learn about cost optimization
*   **[Examples](/docs/examples/bigquery-caching.md)**: See complete BigQuery + caching example
*   **[PostgreSQL](/docs/guides/connections/postgres.md)**: Connect to other databases

([🍪 Cookie Settings](#cookie-settings))

# Google Sheets as Database

**Extension Credit:**

This guide uses the **[gsheets extension](https://duckdb.org/community_extensions/extensions/gsheets.html)** by [archiewood](https://github.com/archiewood) and [mharrisb1](https://github.com/mharrisb1). Thanks to the DuckDB community for this excellent extension that enables seamless Google Sheets integration!

Turn Google Sheets into production-ready REST APIs. Perfect for non-technical teams, rapid prototyping, and collaborative data management - **no database setup required**.

## Why Use Google Sheets with flAPI?

**"The database your marketing team can actually use."**

Google Sheets is the world's most accessible "database":

*   **No setup** - just create a spreadsheet
*   **Collaborative** - multiple people can edit simultaneously
*   **Familiar** - everyone knows how to use spreadsheets
*   **Visual** - see your data, no SQL queries needed
*   **Version history** - built-in audit trail

## Real-World Use Cases

### Content Management for Marketing

Marketing teams manage content without developer intervention:

*   Blog post metadata (titles, authors, tags)
*   Product descriptions and pricing
*   Landing page copy variants
*   Campaign tracking parameters

### Form & Survey Backends

Google Forms → Google Sheets → flAPI → Your App:

*   Customer feedback collection
*   Event registrations
*   Product waitlists
*   Survey responses

### Rapid Prototyping

Build MVP APIs in minutes:

*   No database provisioning
*   No schema migrations
*   Just update the spreadsheet

### Team Collaboration

Non-technical stakeholders contribute directly:

*   Support team manages FAQ data
*   Sales team updates product specs
*   HR maintains team directory

## Installation

The Google Sheets extension requires authentication:

flapi.yaml

```
connections:  my-sheets:    init: |      INSTALL gsheets FROM community;      LOAD gsheets;      -- Authenticate (opens browser for OAuth)      CREATE SECRET (TYPE gsheet);
```

### Alternative: Access Token Authentication

For production deployments, use a service account token:

flapi.yaml

```
connections:  my-sheets:    init: |      INSTALL gsheets FROM community;      LOAD gsheets;      CREATE SECRET (        TYPE gsheet,        PROVIDER access_token,        TOKEN '${GOOGLE_ACCESS_TOKEN}'      );
```

**Get your access token:**

1.  Create a Google Cloud project
2.  Enable Google Sheets API
3.  Create service account credentials
4.  Download JSON key and exchange for an access token

## Configuration Examples

### Basic Configuration

flapi.yaml

```
project-name: sheets-apiconnections:  google-sheets:    init: |      INSTALL gsheets FROM community;      LOAD gsheets;      CREATE SECRET (TYPE gsheet);    properties:      spreadsheet_id: '1A2B3C4D5E6F7G8H9I0J'template:  path: './sqls'  environment-whitelist:    - '^GOOGLE_.*'
```

The `spreadsheet_id` is a [DuckDB gsheets extension property](https://duckdb.org/community_extensions/extensions/gsheets.html) — see upstream docs for the full list.

### Multiple Sheets

flapi.yaml

```
connections:  # Product catalog  products-sheet:    init: |      INSTALL gsheets FROM community;      LOAD gsheets;      CREATE SECRET (TYPE gsheet);    properties:      spreadsheet_id: '1A2B3C4D5E6F7G8H9I0J'      sheet_name: 'Products'  # Customer data  customers-sheet:    init: |      LOAD gsheets;  # Already installed above      CREATE SECRET (TYPE gsheet);    properties:      spreadsheet_id: '9J0I8H7G6F5E4D3C2B1A'      sheet_name: 'Customers'
```

## Example 1: Product Catalog API

### Spreadsheet Structure

| product\_id | name | description | price | in\_stock |
| --- | --- | --- | --- | --- |
| PROD-001 | Widget A | Amazing widget | 29.99 | TRUE |
| PROD-002 | Gadget B | Cool gadget | 49.99 | FALSE |
| PROD-003 | Doohickey C | Useful tool | 19.99 | TRUE |

**Share URL:** `https://docs.google.com/spreadsheets/d/1A2B3C4D5E6F7G8H9I0J/edit`

### SQL Template

sqls/products.sql

```
SELECT  product_id,  name,  description,  price,  in_stockFROM read_gsheet('{{{ conn.spreadsheet_id }}}', sheet = 'Products')WHERE 1=1{{#params.in_stock}}  AND in_stock = '{{{ params.in_stock }}}'{{/params.in_stock}}{{#params.max_price}}  AND price <= {{ params.max_price }}{{/params.max_price}}ORDER BY name
```

### Endpoint Configuration

sqls/products.yaml

```
url-path: /products/method: GETtemplate-source: products.sqlconnection:  - products-sheetrequest:  - field-name: in_stock    field-in: query    description: Filter by stock status    required: false    validators:      - type: enum        allowedValues: ['TRUE', 'FALSE']  - field-name: max_price    field-in: query    description: Maximum price filter    required: false    validators:      - type: int        min: 0        max: 1000000
```

### API Usage

```
# Get all productscurl http://localhost:8080/products/# Get in-stock products under $30curl 'http://localhost:8080/products/?in_stock=TRUE&max_price=30'
```

**Response:**

```
{  "data": [    {      "product_id": "PROD-003",      "name": "Doohickey C",      "description": "Useful tool",      "price": 19.99,      "in_stock": true    }  ]}
```

## Example 2: Blog Post Metadata

Perfect for headless CMS scenarios:

### Spreadsheet

| slug | title | author | published\_date | tags |
| --- | --- | --- | --- | --- |
| hello-world | Hello World | John Doe | 2024-01-15 | tech,intro |
| flapi-launch | Launching flAPI | Jane Smith | 2024-02-01 | product,announcement |

sqls/blog\_posts.sql

```
SELECT  slug,  title,  author,  published_date,  tagsFROM read_gsheet('{{{ conn.spreadsheet_id }}}', sheet = 'Posts')WHERE 1=1{{#params.author}}  AND author = '{{{ params.author }}}'{{/params.author}}{{#params.tag}}  AND tags LIKE '%{{{ params.tag }}}%'{{/params.tag}}ORDER BY published_date DESC
```

sqls/blog\_posts.yaml

```
url-path: /blog/posts/method: GETtemplate-source: blog_posts.sqlconnection:  - my-sheetscache:  enabled: true  table: blog_posts_cache  schedule: 5m   # Refresh every 5 minutes  template-file: blog_cache.sql
```

## Performance Considerations

### Caching is Essential

Google Sheets API has rate limits (100 requests/100 seconds). **Always use caching**:

sqls/products.yaml

```
url-path: /products/method: GETtemplate-source: products.sqlconnection:  - products-sheetcache:  enabled: true  table: products_cache  schedule: 10m   # Refresh every 10 minutes  template-file: products_cache.sql
```

sqls/products\_cache.sql

```
-- Full refresh into the DuckLake cacheINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT * FROM read_gsheet('{{{ conn.spreadsheet_id }}}')
```

**Performance improvement:**

*   Direct: ~500ms per request, rate-limited
*   Cached: **1-10ms per request**, no rate limits

### Optimization Tips

1.  **Use specific sheet names** instead of full URLs
2.  **Read only required columns** (reduces bandwidth)
3.  **Set appropriate cache intervals** (balance freshness vs load)
4.  **Avoid all\_varchar=true** (slows type inference)

```
-- Slow: reads entire sheetSELECT * FROM read_gsheet('1ABC...')-- Fast: specific sheet and columnsSELECT name, priceFROM read_gsheet('1ABC...', sheet = 'Products')
```

## Security Best Practices

### 1\. Use Service Accounts in Production

Never use personal OAuth tokens:

```
connections:  sheets-prod:    init: |      INSTALL gsheets FROM community;      LOAD gsheets;      CREATE SECRET (        TYPE gsheet,        PROVIDER access_token,        TOKEN '${GOOGLE_SERVICE_ACCOUNT_TOKEN}'      );
```

### 2\. Restrict Sheet Permissions

*   Give service account **read-only** access when possible
*   Don't make sheets publicly editable
*   Use specific sheet ranges if supported

### 3\. Validate Input

Always validate request parameters with flAPI's built-in validators:

sqls/products.yaml

```
request:  - field-name: product_id    field-in: query    required: true    validators:      - type: string        regex: '^PROD-[0-9]{3}$'
```

### 4\. Sanitize Data

Triple braces escape strings for SQL safety:

```
WHERE product_id = '{{{ params.product_id }}}'
```

## Troubleshooting

### Authentication Failed

**Error:** `Authentication failed`

```
1. Delete existing secret2. Run flAPI with browser access3. Follow OAuth flow
```

### Rate Limit Exceeded

**Error:** `Quota exceeded for quota metric`

**Solution:** Increase cache intervals:

```
cache:  schedule: 30m   # Every 30 minutes instead of 5
```

### Sheet Not Found

**Error:** `Sheet 'Name' not found`

```
-- Verify sheet name (case-sensitive)FROM read_gsheet('1ABC...', sheet = 'Products')  -- correctFROM read_gsheet('1ABC...', sheet = 'products')  -- wrong
```

### Type Inference Issues

Some columns aren't recognized correctly:

```
-- Force all columns as varchar firstSELECT * FROM read_gsheet('1ABC...', all_varchar = true)-- Then cast explicitlySELECT  CAST(price AS DOUBLE) AS price,  CAST(quantity AS INTEGER) AS quantityFROM ...
```

## Limitations

| Limitation | Impact | Workaround |
| --- | --- | --- |
| **100 requests/100s** | Rate limiting | Use caching (essential) |
| **5M cells max** | Large datasets | Split into multiple sheets |
| **No transactions** | Race conditions | Use append-only patterns |
| **Type inference** | Inconsistent types | Use `all_varchar=true` + cast |
| **OAuth complexity** | Deployment | Use service account tokens |

## When to Use vs When to Avoid

### Use Google Sheets When:

*   Non-technical team needs to manage data
*   Rapid prototyping or MVP
*   Data volume < 10,000 rows
*   Updates are infrequent (< 1/minute)
*   Collaboration is key

### Avoid Google Sheets When:

*   High-frequency writes (> 10/second)
*   Data volume > 100,000 rows
*   Need ACID transactions
*   Sub-50ms latency required
*   Complex relational queries

## Migration Path

**Start with Sheets, migrate later:**

```
# Phase 1: Prototype with Sheetsconnections:  data:    init: |      INSTALL gsheets FROM community;      LOAD gsheets;    properties:      spreadsheet_id: '1ABC...'# Phase 2: Migrate to PostgreSQL (same endpoints!)connections:  data:    init: |      INSTALL postgres;      LOAD postgres;      ATTACH 'host=postgres.example.com dbname=production user=${USER} password=${PASS}' AS data_db (TYPE postgres);    properties:      catalog: data_db
```

Your SQL templates and endpoints stay mostly the same — just point at the new attached database.

## Next Steps

*   **[Google Sheets API Example](/docs/examples/google-sheets-api.md)**: Complete working example
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Essential for production
*   **[Authentication](/docs/endpoints/authentication.md)**: Secure your APIs
*   **[BigQuery](/docs/guides/connections/bigquery.md)**: When you outgrow Sheets
*   **[Vector Search](/docs/guides/connections/vector-search.md)**: Add semantic search to Sheets data

## Additional Resources

*   **[DuckDB GSheets Extension](https://duckdb.org/community_extensions/extensions/gsheets.html)**: Official extension docs
*   **[Google Sheets API](https://developers.google.com/sheets/api)**: API documentation
*   **[Service Accounts](https://cloud.google.com/iam/docs/service-accounts)**: Production authentication

---

**Pro Tip:**

Start every project with Google Sheets. It's the fastest way to validate an idea. You can always migrate to a "real" database later - flAPI makes the transition seamless.

([🍪 Cookie Settings](#cookie-settings))

# ODBC: Universal Database Connector

**Extension Credit:**

This guide uses the **[nanodbc extension](https://duckdb.org/community_extensions/extensions/nanodbc.html)** by the DuckDB community. Thanks to the contributors who made universal database connectivity possible through ODBC!

Connect flAPI to **any database** that provides an ODBC driver - including Oracle, Teradata, DB2, Informix, SAP HANA, and proprietary enterprise systems. Perfect for legacy system integration and databases without native DuckDB extensions.

## Why Use ODBC with flAPI?

**"If it has an ODBC driver, flAPI can connect to it."**

Many enterprise systems don't have modern APIs:

*   **Legacy databases** - Oracle 9i, DB2, Informix
*   **Proprietary systems** - vendor-specific databases
*   **Enterprise software** - SAP HANA, Teradata, Vertica
*   **Mainframe data** - AS/400, z/OS DB2
*   **BI tools** - Microsoft Access, FoxPro

**flAPI + ODBC provides:**

*   Universal compatibility - 1000s of ODBC drivers available
*   No custom code - standard ODBC interface
*   Modern REST APIs - legacy data, modern access
*   Caching layer - speed up slow legacy systems
*   SQL abstraction - hide complex vendor SQL

## Supported Databases (via ODBC)

### Enterprise Databases

*   **Oracle** - all versions (8i through 23c)
*   **IBM DB2** - z/OS, LUW, iSeries (AS/400)
*   **IBM Informix** - all versions
*   **Teradata** - data warehouse platform
*   **SAP HANA** - in-memory database
*   **Vertica** - columnar analytics database

### Proprietary/Legacy

*   **Microsoft Access** - .mdb and .accdb files
*   **FileMaker Pro** - custom business applications
*   **FoxPro** - legacy desktop databases
*   **Sybase ASE** - enterprise database
*   **Progress OpenEdge** - business application platform

### Cloud/Modern (when no native driver)

*   **Amazon Redshift** - via ODBC driver
*   **Azure SQL Database** - alternative to native
*   **Google BigQuery** - alternative to native extension

## Architecture

## Installation

### Step 1: Install ODBC Driver

**On Linux (Ubuntu/Debian):**

```
# Install ODBC managersudo apt-get install unixodbc unixodbc-dev# Example: Install Oracle driversudo apt-get install oracle-instantclient-odbc# Example: Install PostgreSQL driver (for testing)sudo apt-get install odbc-postgresql
```

**On macOS:**

```
brew install unixodbc# Example: PostgreSQL driverbrew install psqlodbc
```

**On Windows:**

Drivers usually come with database client software or download from vendor website.

### Step 2: Configure ODBC DSN

/etc/odbcinst.ini

```
# Define driver location[Oracle ODBC]Description = Oracle ODBC DriverDriver = /usr/lib/oracle/19.3/client64/lib/libsqora.so.19.1[PostgreSQL]Description = PostgreSQL ODBC DriverDriver = /usr/lib/x86_64-linux-gnu/odbc/psqlodbcw.so
```

/etc/odbc.ini

```
# Define data sources[OracleProd]Driver = Oracle ODBCServerName = oracle-prod.company.comPort = 1521Database = PRODDBUID = flapi_readerPWD = ${ORACLE_PASSWORD}[TeradataDW]Driver = TeradataDBCName = teradata-dw.company.comDatabase = ANALYTICSUID = api_userPWD = ${TERADATA_PASSWORD}
```

### Step 3: Configure flAPI

flAPI's ODBC pattern uses `init:` to install/load `nanodbc` and `ATTACH` the DSN as a DuckDB catalog. Then reference the catalog in templates via the connection's `properties`.

flapi.yaml

```
connections:  oracle-legacy:    init: |      INSTALL nanodbc FROM community;      LOAD nanodbc;      ATTACH 'DSN=OracleProd;UID=flapi_reader;PWD=${ORACLE_PASSWORD}' AS odbcdb (TYPE nanodbc);    properties:      catalog: odbcdbtemplate:  environment-whitelist:    - '^ORACLE_.*'    - '^TERADATA_.*'    - '^DB2_.*'
```

The `catalog` property is just a free-form value used in SQL templates as `{{ conn.catalog }}` so you can swap DSNs without editing every SQL file.

## Example 1: Oracle Database

### Configuration

flapi.yaml

```
connections:  oracle-erp:    init: |      INSTALL nanodbc FROM community;      LOAD nanodbc;      ATTACH 'DSN=OracleERP;UID=flapi_reader;PWD=${ORACLE_PASSWORD}' AS oracledb (TYPE nanodbc);    properties:      catalog: oracledb      schema: HR
```

### Query Oracle Table

sqls/employees.sql

```
SELECT  employee_id,  first_name,  last_name,  email,  hire_date,  job_title,  salary,  department_nameFROM {{ conn.catalog }}.{{ conn.schema }}.EMPLOYEESWHERE 1=1{{#params.department}}  AND department_name = '{{{ params.department }}}'{{/params.department}}{{#params.min_salary}}  AND salary >= {{ params.min_salary }}{{/params.min_salary}}ORDER BY hire_date DESC
```

sqls/employees.yaml

```
url-path: /hr/employees/method: GETtemplate-source: employees.sqlconnection:  - oracle-erpcache:  enabled: true  table: oracle_employees  schedule: 6h  template-file: employees_cache.sqlrequest:  - field-name: department    field-in: query    description: Filter by department    required: false    validators:      - type: string        regex: '^[A-Za-z ]{1,50}$'  - field-name: min_salary    field-in: query    description: Minimum salary    required: false    validators:      - type: int        min: 0        max: 100000000
```

## Example 2: IBM DB2 (Mainframe)

### AS/400 Connection

```
connections:  db2-mainframe:    init: |      INSTALL nanodbc FROM community;      LOAD nanodbc;      ATTACH 'Driver={IBM i Access ODBC Driver};System=as400.company.com;UID=${DB2_USER};PWD=${DB2_PASSWORD};DefaultLibraries=PRODLIB;' AS db2db (TYPE nanodbc);    properties:      catalog: db2db      library: PRODLIB
```

### Query Legacy System

sqls/inventory.sql

```
-- Query AS/400 inventory tableSELECT  item_code,  item_description,  quantity_on_hand,  quantity_on_order,  reorder_point,  last_order_dateFROM {{ conn.catalog }}.{{ conn.library }}.INVENTORYWHERE status = 'A'{{#params.low_stock}}  AND quantity_on_hand < reorder_point{{/params.low_stock}}ORDER BY item_code
```

## Example 3: Teradata Data Warehouse

```
connections:  teradata-dw:    init: |      INSTALL nanodbc FROM community;      LOAD nanodbc;      ATTACH 'Driver={Teradata};DBCName=teradata.company.com;Database=ANALYTICS;UID=${TERADATA_USER};PWD=${TERADATA_PASSWORD};CharSet=UTF8;' AS tdwh (TYPE nanodbc);    properties:      catalog: tdwh      schema: ANALYTICS
```

sqls/sales\_analytics.sql

```
SELECT  sale_date,  product_category,  region,  SUM(revenue) AS total_revenue,  SUM(quantity) AS units_sold,  COUNT(DISTINCT customer_id) AS unique_customersFROM {{ conn.catalog }}.{{ conn.schema }}.FACT_SALESWHERE sale_date >= CURRENT_DATE - INTERVAL 90 DAY{{#params.region}}  AND region = '{{{ params.region }}}'{{/params.region}}GROUP BY sale_date, product_category, regionORDER BY sale_date DESC
```

## Performance Optimization

### Problem: ODBC Queries are Slow

Legacy databases can be **very slow** (5-30 seconds per query).

### Solution: Aggressive Caching

```
cache:  enabled: true  table: oracle_employees  schedule: 4h  template-file: oracle_cache.sql
```

sqls/oracle\_cache.sql

```
-- Pull data from Oracle into DuckLake cacheINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT *FROM {{ conn.catalog }}.HR.EMPLOYEESWHERE hire_date >= CURRENT_DATE - INTERVAL 5 YEAR
```

**Performance improvement:**

*   Direct ODBC: 5-30 seconds
*   Cached: **1-10ms** (500-3000x faster)

### Incremental Refresh

sqls/oracle\_incremental\_cache.sql

```
-- Only fetch new/updated recordsINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT *FROM {{ conn.catalog }}.HR.EMPLOYEES{{#cache.previousSnapshotTimestamp}}WHERE last_updated_date > TIMESTAMP '{{ cache.previousSnapshotTimestamp }}'{{/cache.previousSnapshotTimestamp}}
```

## Vendor-Specific SQL Handling

### Oracle-Specific Features

```
-- Oracle syntax: ROWNUM, SYSDATE, NVL — pushed down to the DSNSELECT  employee_id,  NVL(manager_id, 0) AS manager_id,  TO_CHAR(hire_date, 'YYYY-MM-DD') AS hire_dateFROM {{ conn.catalog }}.HR.EMPLOYEESWHERE ROWNUM <= 100
```

### DB2-Specific Features

```
-- DB2 syntax: FETCH FIRST, CURRENT DATESELECT *FROM {{ conn.catalog }}.PRODLIB.ITEMSWHERE created >= CURRENT DATE - 30 DAYSFETCH FIRST 1000 ROWS ONLY
```

### Teradata-Specific Features

```
-- Teradata syntax: TOP, CASTSELECT TOP 1000 *FROM {{ conn.catalog }}.ANALYTICS.SALESWHERE sale_date >= CAST(CURRENT_DATE - 90 AS DATE)
```

## Best Practices

### 1\. Use Read-Only Credentials

```
-- In Oracle: Create read-only userCREATE USER flapi_reader IDENTIFIED BY secure_password;GRANT CONNECT, SELECT ANY TABLE TO flapi_reader;-- Revoke write permissionsREVOKE INSERT, UPDATE, DELETE FROM flapi_reader;
```

### 2\. Limit Data Volume

```
-- BAD: Pull entire table (millions of rows)SELECT * FROM {{ conn.catalog }}.PROD.SALES-- GOOD: Filter at source (only recent data)SELECT * FROM {{ conn.catalog }}.PROD.SALESWHERE sale_date >= CURRENT_DATE - INTERVAL 90 DAY
```

### 3\. Connection Pooling

The nanodbc extension manages its own pool per DuckDB session. To configure pooling, add the relevant options to the ODBC connection string inside the `ATTACH` call (`Pooling=true;Max Pool Size=10`) — see the upstream nanodbc / driver docs for supported options.

### 4\. Test ODBC Connection First

```
# Test ODBC connection before configuring flAPIisql OracleProd flapi_reader password# Should connect successfullySQL> SELECT COUNT(*) FROM HR.EMPLOYEES;
```

## Troubleshooting

### Issue: "Data source not found"

**Error:** `[unixODBC][Driver Manager]Data source name not found`

**Solutions:**

1.  Verify DSN exists in `/etc/odbc.ini`
2.  Check DSN name matches exactly (case-sensitive)
3.  Test with `isql DSN_NAME username password`

### Issue: "Driver not found"

**Error:** `[unixODBC][Driver Manager]Can't open lib`

**Solutions:**

1.  Install ODBC driver package
2.  Update `/etc/odbcinst.ini` with correct driver path
3.  Verify driver file exists: `ls -l /path/to/driver.so`

### Issue: Slow queries

**Problem:** Every query takes 10+ seconds

**Solutions:**

1.  **Enable caching** (most important)
2.  Add indexes in source database
3.  Limit data pulled (WHERE clauses)
4.  Use scheduled batch loads instead of real-time

### Issue: Character encoding problems

**Error:** Garbled text or special characters

**Solution:** Set charset in the connection string passed to `ATTACH`:

```
init: |  INSTALL nanodbc FROM community;  LOAD nanodbc;  ATTACH 'DSN=Oracle;CharSet=UTF8;NCharSet=UTF8;' AS oracledb (TYPE nanodbc);
```

### Issue: Connection timeout

**Error:** `Connection timeout expired`

**Solutions:**

1.  Increase timeout in the connection string: `ConnectionTimeout=60`
2.  Check firewall rules
3.  Verify network connectivity to database
4.  Test direct connection (not through flAPI)

## Security Considerations

### 1\. Never Hardcode Credentials

```
# BADinit: |  ATTACH 'DSN=Oracle;UID=admin;PWD=password123' AS oracledb (TYPE nanodbc);# GOODinit: |  ATTACH 'DSN=Oracle;UID=${DB_USER};PWD=${DB_PASSWORD}' AS oracledb (TYPE nanodbc);
```

### 2\. Use Separate ODBC User

Don't use production/admin accounts:

```
CREATE USER flapi_api_reader IDENTIFIED BY secure_password;GRANT SELECT ON schema.table TO flapi_api_reader;
```

### 3\. Encrypt Connections

```
init: |  ATTACH 'DSN=Oracle;Encrypt=yes;TrustServerCertificate=no;' AS oracledb (TYPE nanodbc);
```

### 4\. IP Whitelisting

Configure database firewall to only allow flAPI server IPs.

## Cost Optimization

### 1\. Reduce Database Load

```
# Infrequent refreshes for static datacache:  enabled: true  table: static_lookup  schedule: 24h
```

### 2\. Minimize Data Transfer

```
-- Only fetch required columnsSELECT id, name, status  -- not SELECT *FROM {{ conn.catalog }}.HR.EMPLOYEES
```

## Migration Examples

### From Oracle to PostgreSQL

```
# Phase 1: ODBC to Oracleconnections:  legacy:    init: |      INSTALL nanodbc FROM community;      LOAD nanodbc;      ATTACH 'DSN=Oracle;UID=${USER};PWD=${PASS}' AS legacy_db (TYPE nanodbc);    properties:      catalog: legacy_db# Phase 2: Migrate to PostgreSQLconnections:  modern:    init: |      INSTALL postgres;      LOAD postgres;      ATTACH 'host=postgres.company.com dbname=prod user=${USER} password=${PASS}' AS legacy_db (TYPE postgres);    properties:      catalog: legacy_db
```

SQL stays mostly the same — flAPI provides abstraction.

## Next Steps

*   **[PostgreSQL](/docs/guides/connections/postgres.md)**: Modern alternative to legacy databases
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Essential for ODBC performance
*   **[Examples](/docs/examples/parquet-api.md)**: See caching patterns
*   **[Deployment](/docs/getting-started/deployment.md)**: Deploy to production

## Additional Resources

*   **[DuckDB nanodbc Extension](https://duckdb.org/community_extensions/extensions/nanodbc.html)**: Official extension docs
*   **[unixODBC Documentation](http://www.unixodbc.org/)**: ODBC manager docs
*   **[ODBC Driver List](https://www.connectionstrings.com/)**: Database-specific connection strings

---

**Perfect for Legacy Systems:**

ODBC is your bridge from legacy to modern. Connect flAPI to that 20-year-old Oracle system, add caching, and suddenly it feels fast and modern. No database migration required!

([🍪 Cookie Settings](#cookie-settings))

# Connecting Data Sources

flAPI leverages DuckDB's extension ecosystem to attach data sources. The ten sources below have first-class examples and dedicated guides in this documentation. Because connections are configured via an arbitrary `init:` SQL block, any additional DuckDB community extension can also be wired up — the list is not exhaustive.

## Basic Configuration

Connections are defined in your `flapi.yaml` configuration file:

```
connections:  connection-name:    init: |      # SQL commands to initialize (load extensions)      INSTALL extension_name;      LOAD extension_name;    properties:      # Connection-specific properties      property1: value1      property2: value2
```

The `init:` block runs once per worker and can install community extensions, attach databases, or create DuckDB secrets — anything DuckDB SQL can do.

## Supported Data Sources (with dedicated guides)

Ten connection types are verified and have a dedicated guide:

### Cloud Data Warehouses

*   **[BigQuery](/docs/guides/connections/bigquery.md)** — Google's cloud data warehouse with DuckLake caching
*   **[Snowflake](/docs/guides/connections/snowflake.md)** — Cloud data platform with cost-optimized access

### Databases & Connectivity

*   **[PostgreSQL](/docs/guides/connections/postgres.md)** — Open-source relational database
*   **[ODBC](/docs/guides/connections/odbc.md)** — Universal database connector (Oracle, Teradata, DB2, and more)

### File Formats

*   **[Parquet](/docs/guides/connections/parquet.md)** — Columnar storage format (local or via httpfs on S3/GCS/Azure)

### Enterprise & BI Systems

*   **[SAP ERP](/docs/guides/connections/sap-erp.md)** — SAP NetWeaver / S/4HANA via the ERPL extension
*   **[SAP BW](/docs/guides/connections/sap-bw.md)** — SAP Business Warehouse via ERPL
*   **[Power BI](/docs/guides/connections/powerbi.md)** — Query Power BI semantic models

### No-Code & Collaborative

*   **[Google Sheets](/docs/guides/connections/google-sheets.md)** — Turn spreadsheets into APIs

### AI / ML

*   **[Vector Search](/docs/guides/connections/vector-search.md)** — Semantic search with the DuckDB VSS extension

## Other Sources via Community Extensions

flAPI doesn't ship integrations beyond the ten above, but **anything you can `INSTALL ... FROM community; LOAD ...;` in DuckDB works inside a connection's `init:` block**. Examples that the community has built and that you can wire up yourself include MySQL, SQLite, Iceberg, Delta, MotherDuck, Excel, and the various `httpfs`\-based file readers. These are unverified by flAPI — you'll need to test them against your version of DuckDB and follow the extension's own docs.

## Quick Start Examples

### Local Parquet File

```
connections:  my-data:    properties:      path: './data/customers.parquet'
```

```
-- In your SQL templateSELECT * FROM '{{{conn.path}}}'
```

### BigQuery (community extension)

```
connections:  bigquery-warehouse:    init: |      INSTALL 'bigquery' FROM community;      LOAD 'bigquery';    properties:      project_id: 'my-project-id'
```

```
SELECT * FROM bigquery_scan('project.dataset.table')
```

### PostgreSQL

```
connections:  postgres-db:    init: |      INSTALL postgres;      LOAD postgres;    properties:      host: localhost      port: 5432      database: mydb      username: ${DB_USER}      password: ${DB_PASSWORD}
```

```
SELECT * FROM postgres_scan('mydb', 'public', 'users')
```

### SQLite via Community Extension

```
connections:  northwind-sqlite:    init: |      INSTALL sqlite;      LOAD sqlite;      ATTACH IF NOT EXISTS './examples/data/northwind.sqlite' AS nw (TYPE sqlite);
```

```
SELECT * FROM nw.customers
```

## Environment Variables

Use environment variables for sensitive data:

```
connections:  secure-db:    properties:      host: ${DB_HOST}      username: ${DB_USER}      password: ${DB_PASSWORD}
```

**Environment whitelist** (in `flapi.yaml`):

```
template:  environment-whitelist:    - '^DB_.*'    - '^GOOGLE_.*'
```

## Multiple Connections

Connect to multiple sources in one project:

```
connections:  # Production warehouse  bigquery-prod:    init: |      INSTALL 'bigquery' FROM community;      LOAD 'bigquery';    properties:      project_id: 'prod-project'  # Reference data  customers-parquet:    properties:      path: './data/customers.parquet'  # Operational database  postgres-ops:    init: |      INSTALL postgres;      LOAD postgres;    properties:      host: ops.example.com      database: operations
```

## Using Connections in Endpoints

Specify which connection to use in your endpoint YAML:

```
# Single connectionurl-path: /customers/template-source: customers.sqlconnection:  - customers-parquet# Multiple connections (join across sources!)url-path: /enriched-orders/template-source: enriched_orders.sqlconnection:  - bigquery-warehouse  - customers-parquet
```

## Why Diverse Data Sources Matter

flAPI's value is making **any data source** accessible via REST APIs and MCP tools. A few examples of why uncommon sources are powerful:

### Google Sheets as a Database

Perfect for non-technical teams, prototyping, or collaborative data management — marketing teams can manage content without touching code, and forms become instant APIs.

### Vector Search for AI

Build RAG (Retrieval Augmented Generation) applications: semantic search over documentation, similar-product recommendations, and AI agents with memory.

### Power BI Integration

Reuse existing BI models without rebuilding them — expose dashboard data as APIs and feed mobile apps from the same logic that powers Power BI reports.

### ODBC for Legacy Systems

Connect to Oracle, Teradata, DB2, Informix, mainframe data sources and proprietary databases — anywhere you have an ODBC driver.

## Next Steps

**Popular Sources:**

*   **[Google Sheets](/docs/guides/connections/google-sheets.md)**: Turn spreadsheets into APIs
*   **[BigQuery](/docs/guides/connections/bigquery.md)**: Connect to Google BigQuery with caching
*   **[PostgreSQL](/docs/guides/connections/postgres.md)**: Connect to PostgreSQL
*   **[Parquet Files](/docs/guides/connections/parquet.md)**: Work with local/cloud files

**AI / ML:**

*   **[Vector Search](/docs/guides/connections/vector-search.md)**: Build RAG applications
*   **[Snowflake](/docs/guides/connections/snowflake.md)**: Cloud data warehouse integration

**Enterprise:**

*   **[Power BI](/docs/guides/connections/powerbi.md)**: Query BI models programmatically
*   **[ODBC](/docs/guides/connections/odbc.md)**: Universal database connector

**Examples:**

*   **[Google Sheets API](/docs/examples/google-sheets-api.md)**: Collaborative data API
*   **[Parquet API](/docs/examples/parquet-api.md)**: Local file APIs
*   **[BigQuery Caching](/docs/examples/bigquery-caching.md)**: Cloud warehouse optimization

([🍪 Cookie Settings](#cookie-settings))

# Working with Parquet Files

Parquet is a columnar storage format that's perfect for analytical workloads. flAPI can query Parquet files directly with no additional setup — DuckDB's native Parquet support makes it incredibly fast.

## Setup

No extensions needed! Just specify the file path as a connection property:

```
# flapi.yamlconnections:  customers-data:    properties:      path: './data/customers.parquet'
```

## Querying Parquet Files

### Single File

```
SELECT *FROM '{{{ conn.path }}}'WHERE active = true{{#params.segment}}  AND market_segment = '{{{ params.segment }}}'{{/params.segment}}
```

### Multiple Files (Glob Pattern)

```
connections:  logs-data:    properties:      path: './data/logs/*.parquet'
```

```
-- Query all matching filesSELECT *FROM '{{{ conn.path }}}'WHERE date >= '2024-01-01'
```

### Directory of Parquet Files

```
connections:  events-data:    properties:      path: './data/events/**/*.parquet'
```

## Complete Example

**File Structure:**

```
project/├── flapi.yaml├── data/│   └── customers.parquet└── sqls/    ├── customers.yaml    └── customers.sql
```

**Configuration** (`flapi.yaml`):

```
project-name: parquet-apitemplate:  path: './sqls'connections:  customers-parquet:    properties:      path: './data/customers.parquet'
```

**Endpoint** (`sqls/customers.yaml`):

```
url-path: /customers/method: GETrequest:  - field-name: id    field-in: query    required: false    validators:      - type: int        min: 1  - field-name: segment    field-in: query    required: false    validators:      - type: enum        allowedValues: [AUTOMOBILE, BUILDING, FURNITURE, HOUSEHOLD, MACHINERY]template-source: customers.sqlconnection:  - customers-parquet
```

**Template** (`sqls/customers.sql`):

```
SELECT  c_custkey AS id,  c_name AS name,  c_mktsegment AS segment,  c_acctbal AS balance,  c_comment AS notesFROM '{{{ conn.path }}}'WHERE 1=1{{#params.id}}  AND c_custkey = {{ params.id }}{{/params.id}}{{#params.segment}}  AND c_mktsegment = '{{{ params.segment }}}'{{/params.segment}}ORDER BY c_acctbal DESCLIMIT 100
```

**Run it:**

```
$ ./flapi -c flapi.yaml# Test API$ curl http://localhost:8080/customers/$ curl 'http://localhost:8080/customers/?segment=AUTOMOBILE'$ curl 'http://localhost:8080/customers/?id=12345'
```

## Why Parquet?

### Fast

DuckDB reads Parquet files incredibly fast thanks to:

*   Columnar storage (only read needed columns)
*   Built-in compression
*   Predicate pushdown
*   Parallel processing

### Portable

*   No database server needed
*   Works with local files, S3, HTTP, etc.
*   Easy to version and distribute

### Cost-Effective

*   No database hosting costs
*   No query costs (unlike BigQuery/Snowflake)
*   Perfect for prototypes and smaller datasets

## Advanced Usage

### S3/Cloud Storage

```
connections:  s3-data:    init: |      INSTALL httpfs;      LOAD httpfs;      SET s3_region='us-east-1';    properties:      path: 's3://my-bucket/data/*.parquet'
```

### HTTP/HTTPS

```
connections:  remote-data:    init: |      INSTALL httpfs;      LOAD httpfs;    properties:      path: 'https://example.com/data/file.parquet'
```

### Partitioned Data

```
connections:  partitioned-data:    properties:      path: './data/year=*/month=*/*.parquet'
```

```
-- DuckDB automatically understands Hive-style partitioningSELECT *FROM read_parquet('{{{ conn.path }}}', hive_partitioning = true)WHERE year = 2024 AND month = 1
```

## Best Practices

### 1\. Use Glob Patterns for Multiple Files

```
# Good: Processes all matching filespath: './data/events/*.parquet'
```

### 2\. Leverage Column Pruning

```
-- Good: Only reads needed columnsSELECT id, name, emailFROM '{{{ conn.path }}}'-- Bad: Reads all columnsSELECT *FROM '{{{ conn.path }}}'
```

### 3\. Use Filters to Reduce I/O

```
-- DuckDB pushes these filters down to the parquet readerSELECT *FROM '{{{ conn.path }}}'WHERE date >= '2024-01-01'  -- Only reads relevant row groups
```

## Next Steps

*   **[Quickstart Guide](/docs/getting-started/quickstart.md)**: Complete tutorial with Parquet
*   **[Examples](/docs/examples/parquet-api.md)**: Full working example
*   **[BigQuery Connection](/docs/guides/connections/bigquery.md)**: Graduate to cloud warehouses

([🍪 Cookie Settings](#cookie-settings))

# Connecting to PostgreSQL

PostgreSQL is a powerful open-source relational database. flAPI can query PostgreSQL databases directly or cache results for high-performance API serving.

## Setup

```
# flapi.yamlconnections:  postgres-db:    init: |      INSTALL postgres;      LOAD postgres;    properties:      host: localhost      port: '5432'      database: mydb      user: '${POSTGRES_USER}'      password: '${POSTGRES_PASSWORD}'
```

Connection `properties` are accessible inside SQL templates as `{{ conn.host }}`, `{{ conn.database }}`, etc.

## Environment Variables

Store sensitive credentials in environment variables:

```
export POSTGRES_USER=myuserexport POSTGRES_PASSWORD=mypassword
```

```
# flapi.yamltemplate:  environment-whitelist:    - '^POSTGRES_.*'
```

## Querying PostgreSQL

The DuckDB Postgres extension typically reads its target via `ATTACH` (recommended) or `postgres_scan` once the connection is attached. Use `init` to attach the database, then reference it in templates.

### Attach Pattern (recommended)

```
connections:  postgres-db:    init: |      INSTALL postgres;      LOAD postgres;      ATTACH IF NOT EXISTS 'host={{ conn.host }} port={{ conn.port }} dbname={{ conn.database }} user={{ conn.user }} password={{ conn.password }}' AS pg (TYPE postgres);    properties:      host: localhost      port: '5432'      database: mydb      user: '${POSTGRES_USER}'      password: '${POSTGRES_PASSWORD}'
```

```
-- Query an attached PostgreSQL tableSELECT *FROM pg.public.usersWHERE active = true{{#params.role}}  AND role = '{{{ params.role }}}'{{/params.role}}
```

## Example: User API

**Endpoint** (`sqls/users.yaml`):

```
url-path: /users/method: GETrequest:  - field-name: role    field-in: query    required: false    validators:      - type: enum        allowedValues: ['admin', 'user', 'guest']  - field-name: active    field-in: query    required: false    validators:      - type: enum        allowedValues: ['true', 'false']template-source: users.sqlconnection:  - postgres-db
```

**Template** (`sqls/users.sql`):

```
SELECT  id,  username,  email,  role,  created_atFROM pg.public.usersWHERE 1=1{{#params.active}}  AND active = {{ params.active }}{{/params.active}}{{#params.role}}  AND role = '{{{ params.role }}}'{{/params.role}}ORDER BY created_at DESCLIMIT 100
```

**Note:**

Boolean and floating-point request parameters arrive as strings. Use an `enum` validator with the allowed string values (`["true", "false"]`) or a `string` validator with a `regex:` constraint. flAPI ships seven validator types: `int`, `string`, `enum`, `email`, `uuid`, `date`, `time`.

## With Caching

For high-traffic APIs, cache PostgreSQL data through DuckLake:

**Cache Template** (`sqls/users_cache.sql`):

```
INSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT  user_id,  username,  email,  role,  last_loginFROM pg.public.usersWHERE active = true
```

**Endpoint configuration**:

```
cache:  enabled: true  table: users_cache  schema: analytics  schedule: 15m  # Refresh every 15 minutes  template-file: users_cache.sql
```

## Connection Pooling

The DuckDB Postgres extension manages its own pool when you `ATTACH` the database. Tune behavior with the upstream extension settings (`pg_pages_per_task`, `pg_use_binary_copy`, etc.) inside `init:` if needed — see the [DuckDB Postgres extension docs](https://duckdb.org/docs/extensions/postgres).

## Next Steps

*   **[BigQuery Connection](/docs/guides/connections/bigquery.md)**: Connect to BigQuery
*   **[Parquet Files](/docs/guides/connections/parquet.md)**: Work with file formats
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Cache PostgreSQL data

([🍪 Cookie Settings](#cookie-settings))

# Power BI Integration

**Extension Credit:**

This guide uses the **[pbix extension](https://duckdb.org/community_extensions/extensions/pbix.html)** by the DuckDB community. Thanks to the contributors who made Power BI file integration possible!

Query Power BI data models, DAX measures, and tabular models programmatically via REST API. Reuse existing BI logic without rebuilding, perfect for mobile apps, custom dashboards, and automated reporting.

## Why Use Power BI with flAPI?

**"Your Power BI logic, now available everywhere."**

Power BI contains valuable business logic:

*   **Reuse DAX measures** - complex calculations already defined
*   **No rebuilding** - leverage existing BI models
*   **Mobile/custom apps** - access BI data outside Power BI
*   **Automated reports** - scheduled data extraction
*   **API for dashboards** - build custom visualizations

**flAPI advantages:**

*   Read Power BI `.pbix` files directly
*   Cache for performance (Power BI can be slow)
*   Add custom authentication layers
*   Mobile-friendly REST APIs
*   MCP tools for AI agents

## Real-World Use Cases

### Mobile App with BI Data

**Problem:** Need sales metrics in mobile app, but Power BI Desktop only **Solution:** flAPI exposes Power BI model as REST API, app queries it

### Automated Reporting

**Problem:** Need to email daily reports from Power BI **Solution:** Scheduled job queries flAPI, generates PDF reports

### Custom Dashboards

**Problem:** Want custom UI, not Power BI's interface **Solution:** Build React/Vue dashboard using flAPI endpoints

### AI with BI Context

**Problem:** AI agent needs company metrics for analysis **Solution:** Expose Power BI as MCP tools for Claude/GPT

## Installation

### Install PBIX Extension

flapi.yaml

```
connections:  powerbi-model:    init: |      INSTALL pbix FROM community;      LOAD pbix;    properties:      pbix_file: './models/sales_model.pbix'
```

The `pbix_file` property is referenced in templates as `{{{ conn.pbix_file }}}`. It is a [DuckDB pbix extension property](https://duckdb.org/community_extensions/extensions/pbix.html) — see upstream docs for the full list of options.

### Prerequisites

1.  **Power BI Desktop** file (`.pbix`)
2.  **Extract to accessible location**
3.  **Read permissions** on file

## Configuration

### Basic Setup

flapi.yaml

```
project-name: powerbi-apiconnections:  sales-model:    init: |      INSTALL pbix FROM community;      LOAD pbix;    properties:      pbix_file: '/data/models/sales_dashboard.pbix'
```

### Multiple Models

```
connections:  # Sales analytics  sales-model:    init: |      INSTALL pbix FROM community;      LOAD pbix;    properties:      pbix_file: '/data/sales_dashboard.pbix'  # Financial reporting  finance-model:    init: |      LOAD pbix;    properties:      pbix_file: '/data/finance_dashboard.pbix'
```

## Example 1: Query Sales Data

### Power BI Model Structure

Typical `.pbix` contains:

*   **Tables**: Sales, Customers, Products, Dates
*   **Relationships**: defined between tables
*   **DAX Measures**: Total Revenue, YoY Growth, etc.
*   **Calculated Columns**: Profit Margin, Customer Segment

### SQL Query

sqls/sales\_summary.sql

```
-- Query tables from Power BI modelSELECT  d.Year,  d.Month,  d.Quarter,  SUM(s.Amount) AS TotalSales,  COUNT(DISTINCT s.CustomerID) AS UniqueCustomers,  COUNT(DISTINCT s.OrderID) AS TotalOrdersFROM pbix_scan('{{{ conn.pbix_file }}}', 'Sales') sJOIN pbix_scan('{{{ conn.pbix_file }}}', 'Dates') d  ON s.DateKey = d.DateKeyWHERE 1=1{{#params.year}}  AND d.Year = {{ params.year }}{{/params.year}}{{#params.quarter}}  AND d.Quarter = {{ params.quarter }}{{/params.quarter}}GROUP BY d.Year, d.Month, d.QuarterORDER BY d.Year DESC, d.Month DESC
```

sqls/sales\_summary.yaml

```
url-path: /sales/summary/method: GETtemplate-source: sales_summary.sqlconnection:  - sales-modelcache:  enabled: true  table: sales_summary_cache  schedule: 6h  template-file: sales_cache.sqlrequest:  - field-name: year    field-in: query    description: Filter by year    required: false    validators:      - type: int        min: 2020        max: 2030  - field-name: quarter    field-in: query    description: Filter by quarter (1-4)    required: false    validators:      - type: int        min: 1        max: 4
```

## Example 2: Customer Segmentation

Leverage Power BI's customer segments:

sqls/customer\_segments.sql

```
SELECT  c.CustomerID,  c.CustomerName,  c.Segment,  c.LifetimeValue,  c.LastPurchaseDate,  COUNT(DISTINCT o.OrderID) AS TotalOrders,  SUM(o.Amount) AS TotalSpentFROM pbix_scan('{{{ conn.pbix_file }}}', 'Customers') cLEFT JOIN pbix_scan('{{{ conn.pbix_file }}}', 'Orders') o  ON c.CustomerID = o.CustomerID{{#params.segment}}WHERE c.Segment = '{{{ params.segment }}}'{{/params.segment}}GROUP BY  c.CustomerID,  c.CustomerName,  c.Segment,  c.LifetimeValue,  c.LastPurchaseDateORDER BY TotalSpent DESC
```

## Example 3: DAX Measures (via Pre-computed Tables)

**Challenge:** DuckDB can't execute DAX directly

**Solution:** Power BI calculates measures → export to tables → query with flAPI

### In Power BI

1.  Create a "Measures Table" with pre-computed metrics
2.  Refresh on schedule
3.  Export `.pbix`

### Query Measures

sqls/kpi\_dashboard.sql

```
SELECT  Period,  TotalRevenue,  RevenueYoY,  GrossMargin,  CustomerChurnRate,  AverageOrderValueFROM pbix_scan('{{{ conn.pbix_file }}}', 'KPI_Measures')WHERE Period >= CURRENT_DATE - INTERVAL 12 MONTHORDER BY Period DESC
```

## Performance Optimization

### Challenge: Reading `.pbix` Files

Power BI files can be large (100s of MB) with complex models.

### Solution 1: Cache Aggressively

```
cache:  enabled: true  table: powerbi_cache  schedule: 6h  template-file: powerbi_cache.sql
```

sqls/powerbi\_cache.sql

```
-- Load model into DuckLake cacheINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT * FROM pbix_scan('{{{ conn.pbix_file }}}', 'Sales')
```

### Solution 2: Export to Parquet

```
# One-time: export Power BI tables to Parquetduckdb -c "  INSTALL pbix FROM community;  LOAD pbix;  COPY (SELECT * FROM pbix_scan('model.pbix', 'Sales'))  TO 'sales.parquet' (FORMAT PARQUET);"
```

Then query Parquet (much faster):

```
SELECT * FROM 'sales.parquet'WHERE Year = 2024
```

### Performance Comparison

| Approach | Query Time | Setup Effort |
| --- | --- | --- |
| **Direct .pbix** | 200-500ms | Low (just reference file) |
| **Cached .pbix** | 1-10ms | Medium (configure caching) |
| **Exported Parquet** | 1-5ms | High (export pipeline) |

**Recommendation:** Start with caching, migrate to Parquet if needed.

## Limitations

### What Works

*   Read all tables from `.pbix`
*   Query relationships (via JOIN)
*   Access calculated columns
*   Filter and aggregate
*   Mix with other data sources

### What Doesn't Work

*   **Execute DAX directly** - no DAX engine in DuckDB
*   **Write back to .pbix** - read-only
*   **Real-time data** - must refresh `.pbix` file
*   **Power BI Service** - only works with `.pbix` files, not cloud datasets

### Workaround for DAX

**Option 1:** Pre-compute measures in Power BI

```
Power BI DAX:YoY Growth = CALCULATE(  [Total Sales],  DATEADD('Date'[Date], -1, YEAR))→ Materialize as calculated column or separate table→ Query with flAPI
```

**Option 2:** Reimplement in SQL

```
SELECT  current_year.Month,  current_year.Sales AS CurrentSales,  prior_year.Sales AS PriorYearSales,  ROUND(    (current_year.Sales - prior_year.Sales) / prior_year.Sales * 100,    2  ) AS YoYGrowthPercentFROM sales_current current_yearLEFT JOIN sales_prior prior_year  ON current_year.Month = prior_year.Month
```

## Best Practices

### 1\. Version Control .pbix Files

```
# Store .pbix in version control or shared storage/data/models/├── sales_v2024.01.pbix├── finance_v2024.01.pbix└── hr_dashboard.pbix
```

### 2\. Automate .pbix Refresh

```
#!/bin/bash# refresh_powerbi.sh - run on schedule# Refresh Power BI Desktop file (requires Windows + Power BI Desktop)powershell -Command "  & 'C:\Program Files\Microsoft Power BI Desktop\bin\PBIDesktop.exe' \  -openFile 'C:\models\sales_dashboard.pbix' \  -refresh"# Copy updated file to flAPI locationcp C:\models\sales_dashboard.pbix /data/models/
```

### 3\. Secure Model Files

Restrict file permissions on the server, and protect the endpoint with one of flAPI's supported auth schemes (`basic`, `jwt`, `bearer`, `oidc`):

```
# sqls/sales_summary.yamlauth:  enabled: true  type: bearer  jwt-secret: '${POWERBI_JWT_SECRET}'
```

## Troubleshooting

### Issue: "Cannot open .pbix file"

**Error:** `Failed to read Power BI file`

**Solutions:**

1.  Verify file path is correct
2.  Check file permissions (flAPI must have read access)
3.  Ensure `.pbix` is not open in Power BI Desktop
4.  Validate file is not corrupted

### Issue: Slow queries

**Problem:** Queries take > 500ms

**Solutions:**

1.  Enable caching (essential)
2.  Export to Parquet for large datasets
3.  Use incremental refresh for large models

### Issue: Tables not found

**Error:** `Table 'Sales' not found in .pbix`

```
# List all tables in .pbixduckdb -c "  INSTALL pbix FROM community;  LOAD pbix;  SELECT * FROM pbix_tables('model.pbix');"
```

### Issue: Outdated data

**Problem:** API returns stale data

**Solution:** Refresh `.pbix` file more frequently

```
# Automate with cron0 */2 * * * /scripts/refresh_powerbi.sh
```

## Migration Path

### Phase 1: Direct .pbix Access

```
connections:  powerbi:    init: |      INSTALL pbix FROM community;      LOAD pbix;    properties:      pbix_file: '/data/sales_dashboard.pbix'
```

### Phase 2: Add Caching

```
cache:  enabled: true  table: powerbi_cache  schedule: 6h  template-file: powerbi_cache.sql
```

### Phase 3: Export to Parquet (for scale)

```
# Export high-traffic tablesduckdb export_to_parquet.sql
```

```
-- export_to_parquet.sqlCOPY (SELECT * FROM pbix_scan('model.pbix', 'Sales'))TO 'sales.parquet';
```

## Next Steps

*   **[BigQuery](/docs/guides/connections/bigquery.md)**: Alternative BI data warehouse
*   **[Snowflake](/docs/guides/connections/snowflake.md)**: Cloud data warehouse
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Optimize performance
*   **[Examples](/docs/examples/parquet-api.md)**: See similar patterns

## Additional Resources

*   **[DuckDB PBIX Extension](https://duckdb.org/community_extensions/extensions/pbix.html)**: Official extension docs
*   **[Power BI Documentation](https://docs.microsoft.com/en-us/power-bi/)**: Microsoft Power BI guide
*   **[DAX Reference](https://dax.guide/)**: DAX function reference

---

**When to Use:**

Power BI integration is perfect when you have **existing BI models** with complex logic that you want to expose via API. Don't rebuild what already works - just expose it with flAPI!

([🍪 Cookie Settings](#cookie-settings))

# SAP BW / BW/4HANA Connection

**Extension Credit:**

This guide uses the **[erpl extension](https://duckdb.org/community_extensions/extensions/erpl.html)** by the DuckDB community. Thanks to the contributors who made SAP BW/4HANA integration possible through RFC connections!

Connect flAPI to SAP BW (Business Warehouse) and BW/4HANA systems using the ERPL extension. Access InfoCubes, DSOs, and BEx queries for analytics APIs.

## Prerequisites

*   SAP NetWeaver RFC SDK installed
*   SAP BW user with query authorizations
*   Network access to BW system

## Quick Start

The ERPL extension is installed from its custom repository and authenticates with a DuckDB `sap_rfc` secret:

```
# flapi.yamlconnections:  sap-bw:    init: |      INSTALL 'erpl' FROM 'http://get.erpl.io';      LOAD 'erpl';      CREATE OR REPLACE PERSISTENT SECRET sap_bw (        TYPE sap_rfc,        ASHOST '${SAP_BW_HOST}',        SYSNR  '${SAP_BW_SYSNR}',        CLIENT '${SAP_BW_CLIENT}',        USER   '${SAP_BW_USER}',        PASSWD '${SAP_BW_PASSWORD}',        LANG   'EN'      );
```

## Connection Parameters

Same as SAP ERP:

| Parameter | Description | Required | Example |
| --- | --- | --- | --- |
| `ASHOST` | SAP BW server hostname | Yes | `bw.company.com` |
| `SYSNR` | System number | Yes | `'00'` |
| `CLIENT` | Client number | Yes | `'100'` |
| `USER` | SAP username | Yes | `${SAP_BW_USER}` |
| `PASSWD` | SAP password | Yes | `${SAP_BW_PASSWORD}` |
| `LANG` | Language | No  | `EN` |

## Reading BW Data

### InfoCube Queries

```
# sqls/sales-analysis.yamlurl-path: /sales/analysis/method: GETrequest:  - field-name: year    field-in: query    required: true    validators:      - type: string        regex: '^\d{4}$'  - field-name: region    field-in: query    required: false    validators:      - type: enum        allowedValues: [EMEA, AMER, APAC]template-source: sales-analysis.sqlconnection:  - sap-bw
```

```
-- sqls/sales-analysis.sql-- Query InfoCube for sales data via the ERPL sap_read_table functionSELECT  "/BIC/OCALYEAR"   AS year,  "/BIC/OCREGION"   AS region,  "/BIC/OCPRODUCT"  AS product,  "/BIC/OCUSTOMER"  AS customer,  "/BIC/OREVENUE"   AS revenue,  "/BIC/OQUANTITY"  AS quantity,  "/BIC/OMARGIN"    AS marginFROM sap_read_table('/BIC/ASALES01')WHERE "/BIC/OCALYEAR" = '{{{ params.year }}}'{{#params.region}}  AND "/BIC/OCREGION" = '{{{ params.region }}}'{{/params.region}}
```

### DSO (DataStore Object) Queries

```
-- sqls/customer-360.sqlSELECT  CUSTOMER_ID,  CUSTOMER_NAME,  SEGMENT,  TOTAL_PURCHASES,  LAST_ORDER_DATE,  LIFETIME_VALUE,  CHURN_RISK_SCOREFROM sap_read_table('/BIC/ACUST360')WHERE ACTIVE_FLAG = 'X'ORDER BY LIFETIME_VALUE DESCLIMIT 1000
```

### ADSO (Advanced DSO in BW/4HANA)

```
-- sqls/realtime-inventory.sqlSELECT  PLANT,  MATERIAL,  STORAGE_LOCATION,  STOCK_QUANTITY,  LAST_UPDATE_TIMESTAMPFROM sap_read_table('/BIC/AINVENT01')WHERE LAST_UPDATE_TIMESTAMP >= CURRENT_TIMESTAMP - INTERVAL 1 HOUR
```

## Common BW Objects

### InfoCubes

Pre-aggregated analytical data:

*   `0SD_C03` - Sales analysis
*   `0PUR_C01` - Purchase analysis
*   `0FI_GL_4` - General ledger
*   `/BIC/A*` - Custom InfoCubes

### DSOs

Detailed transactional data:

*   `0MATERIAL` - Material master
*   `0CUSTOMER` - Customer master
*   `/BIC/A*` - Custom DSOs

### BEx Queries

Access published BEx queries via ERPL (requires RFC function modules).

## Caching BW Data

**Critical for BW**: queries can take 30+ seconds. **Always cache**:

```
# sqls/sales-dashboard.yamlurl-path: /sales/dashboard/method: GETcache:  enabled: true  table: bw_sales_cache  schedule: 12h  template-file: sales_dashboard_cache.sqltemplate-source: sales_dashboard.sqlconnection:  - sap-bw
```

```
-- sqls/sales_dashboard_cache.sql-- Materialize complex BW query results into DuckLakeINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT  c.CALYEAR        AS year,  c.CALMONTH       AS month,  r.REGION_TEXT    AS region,  p.PRODUCT_TEXT   AS product_category,  SUM(f.REVENUE)   AS total_revenue,  SUM(f.QUANTITY)  AS total_quantity,  AVG(f.MARGIN_PCT) AS avg_margin_pct,  COUNT(DISTINCT f.CUSTOMER) AS unique_customersFROM sap_read_table('/BIC/FSALES') fLEFT JOIN sap_read_table('/BIC/PCALENDAR') c  ON f.CALYEAR = c.CALYEAR AND f.CALMONTH = c.CALMONTHLEFT JOIN sap_read_table('/BIC/PREGION') r  ON f.REGION = r.REGIONLEFT JOIN sap_read_table('/BIC/PPRODUCT') p  ON f.PRODUCT = p.PRODUCTWHERE c.CALYEAR >= YEAR(CURRENT_DATE) - 2GROUP BY c.CALYEAR, c.CALMONTH, r.REGION_TEXT, p.PRODUCT_TEXT
```

**Performance:**

| Method | Query Time | API Response |
| --- | --- | --- |
| Direct BW | 30-120 seconds | Timeout |
| flAPI cache | N/A | 1-3ms |

## BW-Specific Considerations

### 1\. Technical Names

BW uses cryptic technical names:

*   InfoCubes: `/BIC/ASALES01`, `0SD_C03`
*   Characteristics: `/BIC/OCUSTOMER`, `0CALMONTH`
*   Key figures: `/BIC/OREVENUE`, `0AMOUNT`

**Find names in:**

*   Transaction RSA1 (Data Warehousing Workbench)
*   Transaction LISTCUBE (view InfoCube data)

### 2\. Naming Conventions

```
/BIC/A*  - Custom InfoCubes/DSOs/BIC/O*  - InfoObject characteristics/BIC/P*  - Master data tables0*       - SAP standard objects
```

### 3\. Date/Time Fields

BW uses specific formats:

*   `CALMONTH`: YYYYMM (e.g., 202401)
*   `CALDAY`: YYYYMMDD (e.g., 20240115)
*   Convert in SQL: `strptime(CALDAY, '%Y%m%d')`

### 4\. Master Data

Join with master data for descriptions:

```
SELECT  f.CUSTOMER,  m.CUSTOMER_NAME,  m.CITY,  m.COUNTRY,  SUM(f.REVENUE) AS revenueFROM sap_read_table('/BIC/FSALES') fLEFT JOIN sap_read_table('/BIC/PCUSTOMER') m  ON f.CUSTOMER = m.CUSTOMERGROUP BY f.CUSTOMER, m.CUSTOMER_NAME, m.CITY, m.COUNTRY
```

## Complete Example

```
# ═══════════════════════════════════════════════════════════════# flapi.yaml - BW connection# ═══════════════════════════════════════════════════════════════project-name: sap-bw-reporting-apitemplate:  path: './sqls'  environment-whitelist:    - '^SAP_BW_.*'    - '^REPORTING_.*'connections:  sap-bw-prod:    init: |      INSTALL 'erpl' FROM 'http://get.erpl.io';      LOAD 'erpl';      CREATE OR REPLACE PERSISTENT SECRET sap_bw_prod (        TYPE sap_rfc,        ASHOST '${SAP_BW_HOST}',        SYSNR  '${SAP_BW_SYSNR}',        CLIENT '${SAP_BW_CLIENT}',        USER   '${SAP_BW_USER}',        PASSWD '${SAP_BW_PASSWORD}',        LANG   'EN'      );# ═══════════════════════════════════════════════════════════════# sqls/revenue-report.yaml# ═══════════════════════════════════════════════════════════════url-path: /reports/revenue/method: GETauth:  enabled: true  type: bearer  jwt-secret: '${REPORTING_JWT_SECRET}'cache:  enabled: true  table: bw_revenue_report  schedule: 24h  template-file: revenue_cache.sqlrequest:  - field-name: year    field-in: query    description: Fiscal year (YYYY)    required: false    default: "2024"    validators:      - type: string        regex: '^\d{4}$'  - field-name: region    field-in: query    description: Region code    required: false    validators:      - type: enum        allowedValues: [EMEA, AMER, APAC]template-source: revenue_report.sqlconnection:  - sap-bw-prod
```

```
-- sqls/revenue_cache.sql-- Daily materialization from BW InfoCubeINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT  strptime(fc.CALMONTH || '01', '%Y%m%d') AS month_date,  SUBSTRING(fc.CALMONTH, 1, 4)            AS year,  SUBSTRING(fc.CALMONTH, 5, 2)            AS month,  reg.REGION_TEXT                         AS region,  pc.PRODUCT_CAT_TEXT                     AS product_category,  cc.COUNTRY_TEXT                         AS country,  SUM(fc.NET_REVENUE)                     AS net_revenue,  SUM(fc.GROSS_REVENUE)                   AS gross_revenue,  SUM(fc.QUANTITY)                        AS quantity_sold,  COUNT(DISTINCT fc.ORDER_NUMBER)         AS order_count,  AVG(fc.MARGIN_PCT)                      AS avg_margin_pctFROM sap_read_table('/BIC/AREVENUE01') fcLEFT JOIN sap_read_table('/BIC/PREGION') reg  ON fc.REGION = reg.REGIONLEFT JOIN sap_read_table('/BIC/PPRODCAT') pc  ON fc.PRODUCT_CAT = pc.PRODUCT_CATLEFT JOIN sap_read_table('/BIC/PCOUNTRY') cc  ON fc.COUNTRY = cc.COUNTRYWHERE SUBSTRING(fc.CALMONTH, 1, 4) >= '2020'  AND fc.RECORD_MODE = 'A'GROUP BY fc.CALMONTH, reg.REGION_TEXT, pc.PRODUCT_CAT_TEXT, cc.COUNTRY_TEXT
```

```
-- sqls/revenue_report.sql-- Fast API queries against the materialized cacheSELECT  year,  month,  month_date,  region,  product_category,  country,  net_revenue,  gross_revenue,  quantity_sold,  order_count,  avg_margin_pctFROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE 1=1{{#params.year}}  AND year = '{{{ params.year }}}'{{/params.year}}{{#params.region}}  AND region = '{{{ params.region }}}'{{/params.region}}ORDER BY month_date DESC, net_revenue DESC
```

## Best Practices

### DO

*   **Cache aggressively** (BW queries are slow)
*   **Refresh overnight** when BW updates
*   **Use master data joins** for readable output
*   **Filter by date ranges** to limit data volume
*   **Test queries in SE16N/LISTCUBE** first
*   **Request BW indexes** on frequently filtered fields

### DON'T

*   Query BW directly without caching
*   Use `SELECT *` on large InfoCubes
*   Expose raw technical names in APIs
*   Query historical data without date filters
*   Run real-time queries against InfoCubes

## Troubleshooting

### No Data Returned

```
-- Check if data existsSELECT COUNT(*) FROM sap_read_table('/BIC/ASALES01')
```

**Common causes:**

*   Wrong technical name
*   No data for filter criteria
*   Need to activate DSO/InfoCube
*   Authorization missing

### Slow Queries

BW query taking > 60 seconds:

**Solutions:**

1.  Enable caching (required)
2.  Add date range filters
3.  Request BW team add indexes
4.  Use aggregated InfoCubes instead of DSOs
5.  Pre-aggregate in cache template

### Authorization Issues

```
Error: RFC_ERROR_AUTHORITY
```

**Required authorizations:**

*   `S_RFC` - RFC authorization
*   `S_RS_COMP` - BW component authorization
*   Query-specific authorizations

Contact SAP Basis/BW team.

## Environment Setup

```
# .envexport SAP_BW_HOST="bw-prod.company.com"export SAP_BW_SYSNR="00"export SAP_BW_CLIENT="100"export SAP_BW_USER="BW_API_USER"export SAP_BW_PASSWORD="secure-password"
```

## Next Steps

*   **[SAP ERP Connection](/docs/guides/connections/sap-erp.md)**: Connect to SAP ERP/S4HANA
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Essential for BW
*   **[BigQuery Connection](/docs/guides/connections/bigquery.md)**: Alternative cloud warehouse

([🍪 Cookie Settings](#cookie-settings))

# SAP ERP Connection

**Extension Credit:**

This guide uses the **[erpl extension](https://duckdb.org/community_extensions/extensions/erpl.html)** by the DuckDB community. Thanks to the contributors who made SAP ERP integration possible through RFC connections!

Connect flAPI to SAP ERP systems (ECC, S/4HANA) using DuckDB's ERPL extension. Extract data from tables, run function modules, and execute ABAP queries directly.

## Prerequisites

*   SAP NetWeaver RFC SDK installed
*   SAP user with appropriate authorizations
*   Network access to SAP system

## Quick Start

The ERPL extension is installed via its custom repository and authenticates with a DuckDB `sap_rfc` secret:

```
# flapi.yamlconnections:  sap-erp:    init: |      INSTALL 'erpl' FROM 'http://get.erpl.io';      LOAD 'erpl';      CREATE OR REPLACE PERSISTENT SECRET sap_erp (        TYPE sap_rfc,        ASHOST '${SAP_ASHOST}',        SYSNR  '${SAP_SYSNR}',        CLIENT '${SAP_CLIENT}',        USER   '${SAP_USER}',        PASSWD '${SAP_PASSWORD}',        LANG   'EN'      );
```

The secret name (`sap_erp`) is referenced implicitly by the ERPL extension functions like `sap_read_table()`.

## Connection Parameters

| Parameter | Description | Required | Example |
| --- | --- | --- | --- |
| `ASHOST` | SAP application server hostname | Yes | `sap.company.com` |
| `SYSNR` | System number | Yes | `'00'` |
| `CLIENT` | Client number | Yes | `'100'` |
| `USER` | SAP username | Yes | `${SAP_USER}` |
| `PASSWD` | SAP password | Yes | `${SAP_PASSWORD}` |
| `LANG` | Language (EN, DE, etc.) | No  | `EN` |

## Reading SAP Tables

### Basic Table Read

```
# sqls/materials.yamlurl-path: /materials/method: GETrequest:  - field-name: plant    field-in: query    description: Plant code    required: false    validators:      - type: string        regex: '^[A-Z0-9]{1,4}$'template-source: materials.sqlconnection:  - sap-erp
```

```
-- sqls/materials.sqlSELECT  MATNR AS material_number,  WERKS AS plant,  LGORT AS storage_location,  LABST AS stock_quantityFROM sap_read_table('MARD')WHERE 1=1{{#params.plant}}  AND WERKS = '{{{ params.plant }}}'{{/params.plant}}
```

### Joining SAP Tables

```
-- sqls/sales-orders.sqlSELECT  h.VBELN AS sales_order,  h.ERDAT AS creation_date,  h.KUNNR AS customer_number,  c.NAME1 AS customer_name,  i.MATNR AS material,  i.KWMENG AS quantity,  i.NETWR AS net_valueFROM sap_read_table('VBAK') hLEFT JOIN sap_read_table('KNA1') c  ON h.KUNNR = c.KUNNRLEFT JOIN sap_read_table('VBAP') i  ON h.VBELN = i.VBELNWHERE h.ERDAT >= '20240101'ORDER BY h.ERDAT DESC
```

## Common SAP Tables

### Materials Management (MM)

*   `MARA` - Material master (general data)
*   `MAKT` - Material descriptions
*   `MARD` - Storage location data
*   `MARC` - Plant data for materials

### Sales & Distribution (SD)

*   `VBAK` - Sales document header
*   `VBAP` - Sales document items
*   `KNA1` - Customer master (general)
*   `KNVV` - Customer master (sales)

### Finance (FI)

*   `BKPF` - Accounting document header
*   `BSEG` - Accounting document items
*   `SKA1` - G/L account master

### Purchasing (MM-PUR)

*   `EKKO` - Purchase order header
*   `EKPO` - Purchase order items
*   `LFA1` - Vendor master

## Caching SAP Data

SAP tables are large and slow. **Always use caching**:

```
# sqls/materials.yamlurl-path: /materials/method: GETcache:  enabled: true  table: sap_materials_cache  schedule: 6h  template-file: materials_cache.sqltemplate-source: materials.sqlconnection:  - sap-erp
```

```
-- sqls/materials_cache.sql-- Load all active materials into the DuckLake cacheINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT  m.MATNR AS material_number,  t.MAKTX AS material_description,  m.MTART AS material_type,  m.MATKL AS material_group,  m.MEINS AS base_unit,  s.WERKS AS plant,  s.LABST AS stock_quantityFROM sap_read_table('MARA') mJOIN sap_read_table('MAKT') t  ON m.MATNR = t.MATNR  AND t.SPRAS = 'E'JOIN sap_read_table('MARD') s  ON m.MATNR = s.MATNRWHERE m.LVORM IS NULL
```

**Performance comparison:**

| Method | Latency | Cost |
| --- | --- | --- |
| Direct SAP query | 5-30 seconds | High RFC load |
| flAPI cache | 1-5ms | Minimal (scheduled refresh) |

## Security Best Practices

### DO

```
# Use environment variables in the SECRETconnections:  sap-erp:    init: |      INSTALL 'erpl' FROM 'http://get.erpl.io';      LOAD 'erpl';      CREATE OR REPLACE PERSISTENT SECRET sap_erp (        TYPE sap_rfc,        ASHOST '${SAP_ASHOST}',        SYSNR  '${SAP_SYSNR}',        CLIENT '${SAP_CLIENT}',        USER   '${SAP_USER}',        PASSWD '${SAP_PASSWORD}',        LANG   'EN'      );
```

### Restrict the endpoint with authentication

```
# sqls/materials.yamlauth:  enabled: true  type: bearer  jwt-secret: '${SAP_JWT_SECRET}'
```

flAPI endpoint auth uses a single `type` string. Supported values: `basic`, `jwt`, `bearer`, `oidc`.

### DON'T

```
# Never hardcode credentials inside the secretinit: |  CREATE OR REPLACE PERSISTENT SECRET sap_erp (    TYPE sap_rfc,    USER   'JOHNDOE',          -- BAD    PASSWD 'Password123'       -- BAD  );
```

### Environment Setup

```
# .env (never commit)export SAP_USER="RFC_API_USER"export SAP_PASSWORD="secure-password"export SAP_ASHOST="sap-prod.company.com"export SAP_SYSNR="00"export SAP_CLIENT="100"
```

```
# flapi.yamltemplate:  path: './sqls'  environment-whitelist:    - '^SAP_.*'
```

## Complete Example

```
# ═══════════════════════════════════════════════════════════════# flapi.yaml# ═══════════════════════════════════════════════════════════════project-name: sap-inventory-apitemplate:  path: './sqls'  environment-whitelist:    - '^SAP_.*'connections:  sap-prod:    init: |      INSTALL 'erpl' FROM 'http://get.erpl.io';      LOAD 'erpl';      CREATE OR REPLACE PERSISTENT SECRET sap_prod (        TYPE sap_rfc,        ASHOST '${SAP_ASHOST}',        SYSNR  '${SAP_SYSNR}',        CLIENT '${SAP_CLIENT}',        USER   '${SAP_USER}',        PASSWD '${SAP_PASSWORD}',        LANG   'EN'      );# ═══════════════════════════════════════════════════════════════# sqls/inventory.yaml# ═══════════════════════════════════════════════════════════════url-path: /inventory/method: GETauth:  enabled: true  type: bearer  jwt-secret: '${INVENTORY_JWT_SECRET}'cache:  enabled: true  table: sap_inventory  schedule: 1h  template-file: inventory_cache.sqlrequest:  - field-name: plant    field-in: query    description: Plant code (e.g., 1000)    required: false    validators:      - type: string        regex: '^[A-Z0-9]{1,4}$'  - field-name: material_type    field-in: query    description: Material type (e.g., FERT, HALB)    required: false    validators:      - type: enum        allowedValues: [FERT, HALB, ROH, HAWA]template-source: inventory.sqlconnection:  - sap-prod
```

```
-- sqls/inventory_cache.sql-- Materialize inventory dataINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT  m.MATNR  AS material,  t.MAKTX  AS description,  m.MTART  AS material_type,  s.WERKS  AS plant,  s.LGORT  AS storage_location,  s.LABST  AS unrestricted_stock,  s.INSME  AS quality_inspection_stock,  s.SPEME  AS blocked_stock,  p.STPRS  AS standard_price,  p.PEINH  AS price_unitFROM sap_read_table('MARA') mJOIN sap_read_table('MAKT') t  ON m.MATNR = t.MATNR AND t.SPRAS = 'E'JOIN sap_read_table('MARD') s  ON m.MATNR = s.MATNRJOIN sap_read_table('MBEW') p  ON m.MATNR = p.MATNR AND s.WERKS = p.BWKEYWHERE m.LVORM IS NULL
```

```
-- sqls/inventory.sql-- Fast queries against the cacheSELECT *FROM {{cache.catalog}}.{{cache.schema}}.{{cache.table}}WHERE 1=1{{#params.plant}}  AND plant = '{{{ params.plant }}}'{{/params.plant}}{{#params.material_type}}  AND material_type = '{{{ params.material_type }}}'{{/params.material_type}}ORDER BY unrestricted_stock DESC
```

## Troubleshooting

### Connection Failed

```
Error: RFC_ERROR_COMMUNICATION - Connection to SAP system failed
```

**Solutions:**

*   Verify `ASHOST` and `SYSNR` are correct
*   Check network connectivity: `ping sap.company.com`
*   Ensure SAP NetWeaver RFC SDK is installed
*   Verify firewall allows RFC connections (port 33XX)

### Authorization Errors

```
Error: RFC_ERROR_AUTHORITY - User lacks authorization
```

**Solutions:**

*   Ensure user has RFC authorizations
*   Check transaction authorizations (S\_RFC, S\_TABU\_NAM)
*   Contact SAP Basis team for access

### Slow Queries

SAP queries taking > 10 seconds:

**Solutions:**

*   Enable caching (6h-24h refresh)
*   Use indexes on SAP tables
*   Limit date ranges
*   Avoid `SELECT *`
*   Don't query SAP directly for high-frequency APIs

## Next Steps

*   **[SAP BW Connection](/docs/guides/connections/sap-bw.md)**: Connect to SAP BW/4HANA
*   **[Caching Setup](/docs/guides/caching/setup.md)**: Optimize performance
*   **[Authentication](/docs/endpoints/authentication.md)**: Secure your API

([🍪 Cookie Settings](#cookie-settings))

# Snowflake Integration

**Extension Credit:**

This guide uses the **[snowflake extension](https://duckdb.org/community_extensions/extensions/snowflake.html)** by the DuckDB community. Thanks to the contributors who made Snowflake integration possible!

Turn your Snowflake data warehouse into a high-performance REST API with **99%+ cost savings** through intelligent caching. Perfect for analytics APIs, dashboards, and customer-facing data products.

## Why Use Snowflake with flAPI?

**"The Snowflake caching layer that pays for itself in week 1."**

Snowflake is powerful but **expensive** for high-frequency access:

*   **$2/hour minimum** for smallest warehouse
*   **Cold start delays** (5-15 seconds)
*   **Compute costs** scale with query complexity
*   **Per-query billing** adds up fast

**flAPI solves this:**

*   **99%+ cost reduction** through caching
*   **1-10ms response time** (vs 500ms-5s)
*   **No warehouse always-on costs**
*   **Scheduled refreshes** during off-peak hours
*   **DuckDB-powered queries** (faster for small-medium datasets)

## Real-World Use Cases

### Customer Analytics Dashboard

**Problem:** Dashboard queries hit Snowflake 10,000x/day = $1,500/month **Solution:** Cache refreshes 4x/day = **$15/month** (100x savings)

### Internal Reporting API

**Problem:** Sales team's BI tool hammers Snowflake constantly **Solution:** flAPI cache serves instant results, refresh hourly

### Mobile App Analytics

**Problem:** Every app open = expensive Snowflake query **Solution:** Pre-computed aggregates cached, sub-10ms to users

### AI Agent Data Access

**Problem:** LLMs need fast access to company data **Solution:** flAPI + MCP provides instant Snowflake access for AI

## Architecture

## Installation

### Prerequisites

1.  **Snowflake Account** with credentials
2.  **DuckDB Snowflake Extension**

### Extension Setup

The DuckDB snowflake extension authenticates via a DuckDB `SNOWFLAKE` secret created during `init:`. flAPI connection `properties` are exposed as `{{ conn.* }}` so they can be referenced in templates and the init block.

flapi.yaml

```
connections:  snowflake-prod:    init: |      INSTALL snowflake FROM community;      LOAD snowflake;      CREATE OR REPLACE PERSISTENT SECRET snowflake_prod (        TYPE SNOWFLAKE,        ACCOUNT '${SNOWFLAKE_ACCOUNT}',        USER '${SNOWFLAKE_USER}',        PASSWORD '${SNOWFLAKE_PASSWORD}',        WAREHOUSE '${SNOWFLAKE_WAREHOUSE}',        DATABASE '${SNOWFLAKE_DATABASE}',        SCHEMA '${SNOWFLAKE_SCHEMA}'      );    properties:      database: 'ANALYTICS'      schema: 'PUBLIC'template:  environment-whitelist:    - '^SNOWFLAKE_.*'
```

### Environment Variables

.env

```
SNOWFLAKE_ACCOUNT=xy12345.us-east-1SNOWFLAKE_USER=flapi_readerSNOWFLAKE_PASSWORD=your-password-hereSNOWFLAKE_WAREHOUSE=COMPUTE_WHSNOWFLAKE_DATABASE=ANALYTICSSNOWFLAKE_SCHEMA=PUBLIC
```

## Basic Configuration

### Simple Query

sqls/daily\_sales.sql

```
SELECT  sale_date,  product_category,  SUM(revenue) AS total_revenue,  COUNT(DISTINCT customer_id) AS unique_customersFROM snowflake_scan('{{ conn.database }}.{{ conn.schema }}.SALES')WHERE sale_date >= CURRENT_DATE - INTERVAL 30 DAY{{#params.category}}  AND product_category = '{{{ params.category }}}'{{/params.category}}GROUP BY sale_date, product_categoryORDER BY sale_date DESC
```

sqls/daily\_sales.yaml

```
url-path: /sales/daily/method: GETtemplate-source: daily_sales.sqlconnection:  - snowflake-prodcache:  enabled: true  table: daily_sales_cache  schedule: 6h  template-file: daily_sales_cache.sqlrequest:  - field-name: category    field-in: query    description: Filter by product category    required: false    validators:      - type: string        regex: '^[A-Za-z _-]{1,50}$'
```

### Cache Template

sqls/daily\_sales\_cache.sql

```
-- Full refresh: pull data from Snowflake into the DuckLake cacheINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT  sale_date,  product_category,  revenue,  customer_idFROM snowflake_scan('{{ conn.database }}.{{ conn.schema }}.SALES')WHERE sale_date >= CURRENT_DATE - INTERVAL 30 DAY
```

## Cost Optimization Strategies

### Strategy 1: Scheduled Refreshes (best for most use cases)

sqls/analytics.yaml

```
url-path: /analytics/revenue/method: GETtemplate-source: revenue.sqlconnection:  - snowflake-prodcache:  enabled: true  table: revenue_cache  # Refresh on a fixed cadence  schedule: 6h  template-file: revenue_cache.sql
```

**Cost comparison (10K API calls/day):**

*   Direct Snowflake: ~$1,500/month
*   Cached (4x refresh): ~$15/month
*   **Savings: 99%**

### Strategy 2: Incremental Refresh

sqls/orders\_cache.sql

```
-- Append only new/updated recordsINSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT *FROM snowflake_scan('{{ conn.database }}.{{ conn.schema }}.ORDERS'){{#cache.previousSnapshotTimestamp}}WHERE updated_at > TIMESTAMP '{{ cache.previousSnapshotTimestamp }}'{{/cache.previousSnapshotTimestamp}}
```

```
cache:  enabled: true  table: orders_cache  schedule: 15m  primary-key: [id]  cursor:    column: updated_at    type: timestamp
```

### Strategy 3: Tiered Caching

```
# Hot data: refresh frequently# sqls/sales_today.yamlcache:  enabled: true  table: sales_today  schedule: 5m# Warm data: less frequent# sqls/sales_monthly.yamlcache:  enabled: true  table: sales_monthly  schedule: 6h# Cold data: rarely updated# sqls/sales_yearly.yamlcache:  enabled: true  table: sales_yearly  schedule: 7d
```

## Example: Real-Time Sales Dashboard

### Aggregated Metrics

sqls/sales\_metrics.sql

```
WITH daily_stats AS (  SELECT    sale_date,    SUM(revenue) AS daily_revenue,    COUNT(DISTINCT order_id) AS order_count,    COUNT(DISTINCT customer_id) AS customer_count,    AVG(revenue) AS avg_order_value  FROM snowflake_scan('{{ conn.database }}.{{ conn.schema }}.SALES')  WHERE sale_date >= CURRENT_DATE - INTERVAL 90 DAY  GROUP BY sale_date)SELECT  sale_date,  daily_revenue,  order_count,  customer_count,  ROUND(avg_order_value, 2) AS avg_order_value,  ROUND(AVG(daily_revenue) OVER (    ORDER BY sale_date    ROWS BETWEEN 6 PRECEDING AND CURRENT ROW  ), 2) AS revenue_7day_avgFROM daily_statsORDER BY sale_date DESCLIMIT {{#params.limit}}{{ params.limit }}{{/params.limit}}{{^params.limit}}90{{/params.limit}}
```

sqls/sales\_metrics.yaml

```
url-path: /sales/metrics/method: GETtemplate-source: sales_metrics.sqlconnection:  - snowflake-prodcache:  enabled: true  table: sales_metrics_cache  schedule: 10m  template-file: sales_metrics_cache.sqlrequest:  - field-name: limit    field-in: query    description: Number of days to return    required: false    validators:      - type: int        min: 1        max: 365
```

### Customer Segmentation

sqls/customer\_segments.sql

```
WITH customer_lifetime AS (  SELECT    customer_id,    COUNT(DISTINCT order_id) AS total_orders,    SUM(revenue) AS lifetime_value,    MIN(sale_date) AS first_purchase,    MAX(sale_date) AS last_purchase,    date_diff('day', MAX(sale_date), CURRENT_DATE) AS days_since_last_purchase  FROM snowflake_scan('{{ conn.database }}.{{ conn.schema }}.SALES')  GROUP BY customer_id)SELECT  customer_id,  total_orders,  ROUND(lifetime_value, 2) AS lifetime_value,  first_purchase,  last_purchase,  days_since_last_purchase,  CASE    WHEN days_since_last_purchase <= 30 AND total_orders >= 5 THEN 'VIP Active'    WHEN days_since_last_purchase <= 30 THEN 'Active'    WHEN days_since_last_purchase <= 90 THEN 'At Risk'    WHEN days_since_last_purchase <= 180 THEN 'Dormant'    ELSE 'Lost'  END AS segmentFROM customer_lifetime{{#params.segment}}WHERE segment = '{{{ params.segment }}}'{{/params.segment}}ORDER BY lifetime_value DESC
```

## Performance Comparison

### Direct Snowflake Query

```
Query time: 500ms - 5 secondsCost per query: ~$0.0110K queries/day: $100/day = $3,000/monthCold start penalty: 5-15 seconds
```

### flAPI Cached Query

```
Query time: 1-10ms (50-500x faster)Cost per query: $0 (cached)10K queries/day: $0Cache refresh (4x/day): $0.40/day = $12/monthSavings: 99.6%
```

## Best Practices

### 1\. Use Smallest Warehouse for Refreshes

```
export SNOWFLAKE_WAREHOUSE='XSMALL_WH'  # Not 'LARGE_WH'
```

**Cost difference:**

*   X-Small: $2/hour
*   Large: $32/hour
*   **Savings: 16x**

### 2\. Schedule Refreshes During Off-Peak

```
cache:  # Off-peak refresh once every 6 hours  schedule: 6h
```

### 3\. Materialize Complex Joins in Snowflake

```
-- Create materialized view in SnowflakeCREATE MATERIALIZED VIEW ANALYTICS.PUBLIC.SALES_ENRICHED ASSELECT  s.*,  c.customer_name,  c.customer_segment,  p.product_name,  p.product_categoryFROM SALES sJOIN CUSTOMERS c ON s.customer_id = c.idJOIN PRODUCTS p ON s.product_id = p.id;-- Then query the view (faster, cheaper)SELECT * FROM snowflake_scan('ANALYTICS.PUBLIC.SALES_ENRICHED')
```

### 4\. Partition by Time

```
WHERE sale_date >= CURRENT_DATE - INTERVAL 90 DAY
```

### 5\. Use Column Pruning

```
-- Bad: pulls all columnsSELECT * FROM snowflake_scan('ANALYTICS.PUBLIC.SALES')-- Good: only needed columns (faster, cheaper)SELECT  sale_date,  customer_id,  revenueFROM snowflake_scan('ANALYTICS.PUBLIC.SALES')
```

## Security

### Read-Only User

Create a dedicated read-only user in Snowflake:

```
-- In SnowflakeCREATE ROLE FLAPI_READER;GRANT USAGE ON WAREHOUSE COMPUTE_WH TO ROLE FLAPI_READER;GRANT USAGE ON DATABASE ANALYTICS TO ROLE FLAPI_READER;GRANT USAGE ON SCHEMA ANALYTICS.PUBLIC TO ROLE FLAPI_READER;GRANT SELECT ON ALL TABLES IN SCHEMA ANALYTICS.PUBLIC TO ROLE FLAPI_READER;GRANT SELECT ON FUTURE TABLES IN SCHEMA ANALYTICS.PUBLIC TO ROLE FLAPI_READER;CREATE USER flapi_reader  PASSWORD='...'  DEFAULT_ROLE=FLAPI_READER;GRANT ROLE FLAPI_READER TO USER flapi_reader;
```

### Credential Management

```
connections:  snowflake-prod:    init: |      INSTALL snowflake FROM community;      LOAD snowflake;      CREATE OR REPLACE PERSISTENT SECRET snowflake_prod (        TYPE SNOWFLAKE,        ACCOUNT '${SNOWFLAKE_ACCOUNT}',        USER '${SNOWFLAKE_USER}',        PASSWORD '${SNOWFLAKE_PASSWORD}',        ROLE 'FLAPI_READER'      );
```

### Network Security

Use Snowflake network policies:

```
-- In Snowflake: restrict IP accessCREATE NETWORK POLICY flapi_access  ALLOWED_IP_LIST = ('203.0.113.10/32', '198.51.100.20/32');ALTER USER flapi_reader SET NETWORK_POLICY = flapi_access;
```

## Troubleshooting

### Issue: "Authentication failed"

```
# Test connection with the DuckDB CLIexport SNOWFLAKE_ACCOUNT="xy12345.us-east-1"export SNOWFLAKE_USER="flapi_reader"export SNOWFLAKE_PASSWORD="your-password"duckdb -c "  INSTALL snowflake FROM community;  LOAD snowflake;  CREATE OR REPLACE PERSISTENT SECRET sf (    TYPE SNOWFLAKE,    ACCOUNT  '${SNOWFLAKE_ACCOUNT}',    USER     '${SNOWFLAKE_USER}',    PASSWORD '${SNOWFLAKE_PASSWORD}'  );  SELECT * FROM snowflake_scan('ANALYTICS.PUBLIC.SALES') LIMIT 10;"
```

### Issue: Slow cache refreshes

**Problem:** Cache refresh takes 10+ minutes

**Solutions:**

1.  Use materialized views in Snowflake
2.  Partition by date (only recent data)
3.  Use a larger warehouse for the refresh (balance cost vs speed)
4.  Consider an incremental cursor-based refresh

### Issue: High Snowflake costs despite caching

Tune your refresh cadence:

```
# Too frequent (expensive)cache:  schedule: 1m  # every minute = $$$# Optimal (cheap)cache:  schedule: 6h
```

## Cost Calculator

**Your scenario:**

*   API calls: 10,000/day
*   Cache refreshes: 4/day
*   Refresh duration: 5 minutes each
*   Warehouse: X-Small ($2/hour)

**Direct Snowflake:**

```
10,000 calls × $0.01 per query = $100/day$100 × 30 days = $3,000/month
```

**flAPI Cached:**

```
4 refreshes/day × 5 min × $2/hour = $0.67/day$0.67 × 30 days = $20/monthSavings: $2,980/month (99.3%)
```

## Migration from Direct Access

### Phase 1: Add flAPI Layer

```
connections:  snowflake-prod:    init: |      INSTALL snowflake FROM community;      LOAD snowflake;      CREATE OR REPLACE PERSISTENT SECRET snowflake_prod (        TYPE SNOWFLAKE,        ACCOUNT '${SNOWFLAKE_ACCOUNT}',        USER '${SNOWFLAKE_USER}',        PASSWORD '${SNOWFLAKE_PASSWORD}'      );
```

### Phase 2: Introduce Caching

```
cache:  enabled: true  table: hot_data_cache  schedule: 10m  template-file: hot_data_cache.sql
```

### Phase 3: Monitor Costs

```
-- In Snowflake: check query costsSELECT  query_text,  COUNT(*) AS execution_count,  AVG(total_elapsed_time) / 1000 AS avg_seconds,  AVG(credits_used_cloud_services) AS avg_creditsFROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORYWHERE query_text LIKE '%flapi%'  AND start_time >= DATEADD(day, -7, CURRENT_TIMESTAMP)GROUP BY query_textORDER BY avg_credits DESC;
```

## Next Steps

*   **[Caching Strategy](/docs/concepts/caching-strategy.md)**: Understand optimization techniques
*   **[BigQuery](/docs/guides/connections/bigquery.md)**: Alternative cloud warehouse
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Master dynamic queries
*   **[Examples](/docs/examples/bigquery-caching.md)**: See caching in action
*   **[Deployment](/docs/getting-started/deployment.md)**: Deploy to production

## Additional Resources

*   **[DuckDB Snowflake Extension](https://duckdb.org/community_extensions/extensions/snowflake.html)**: Official extension docs
*   **[Snowflake Documentation](https://docs.snowflake.com/)**: Complete Snowflake reference
*   **[Snowflake Cost Optimization](https://docs.snowflake.com/en/user-guide/cost-understanding-compute)**: Official cost guide

---

**Cost Savings:**

One flAPI user saved **$47,000/year** by caching their customer dashboard queries. The cache refreshes 6x/day, users get instant responses, and Snowflake costs dropped by 98%. ROI payback: 3 days.

([🍪 Cookie Settings](#cookie-settings))

# Vector Search for AI Applications

**Extension Credit:**

This guide uses the **[VSS (Vector Similarity Search)](https://duckdb.org/docs/stable/core_extensions/vss.html)** core extension and **[faiss extension](https://duckdb.org/community_extensions/extensions/faiss.html)** by the DuckDB community. Thanks to the contributors who made vector search accessible in DuckDB!

Build production-ready semantic search and RAG (Retrieval Augmented Generation) APIs using DuckDB's vector extensions. Perfect for AI agents, documentation search, recommendation engines, and knowledge bases.

## Why Vector Search with flAPI?

**"RAG as a service, deployed in minutes."**

Modern AI applications need semantic search:

*   **RAG for LLMs** - give AI context from your data
*   **Semantic search** - find by meaning, not keywords
*   **Knowledge bases** - intelligent document retrieval
*   **Recommendations** - similar products, content, users
*   **Classification** - categorize by similarity

**flAPI makes it production-ready:**

*   Sub-10ms queries with vector indexes
*   SQL-native vector operations (no new query language)
*   REST API + MCP for AI agents
*   Hybrid search (combine vectors + keywords + filters)
*   Multiple backends (Faiss, VSS, built-in)

## Vector Search Options

DuckDB provides two powerful vector extensions:

### Option 1: VSS (Vector Similarity Search)

**Best for: general purpose, simpler setup**

```
init: |  INSTALL vss;  LOAD vss;
```

*   Native DuckDB core extension
*   HNSW index support
*   Good for < 1M vectors
*   Simpler API

### Option 2: Faiss

**Best for: large scale, high performance**

```
init: |  INSTALL faiss FROM community;  LOAD faiss;
```

*   Facebook's battle-tested library
*   Multiple index types (IVF, HNSW, Flat)
*   Optimized for 10M+ vectors
*   Advanced quantization

## Real-World Use Cases

### RAG for Customer Support

**Scenario:** AI chatbot needs to find relevant help articles

```
User: "How do I reset my password?"→ Vector search finds top 3 relevant articles→ LLM generates answer using context
```

### Documentation Search

**Scenario:** Semantic code search across your codebase

```
Query: "authentication middleware"→ Finds auth-related functions by meaning→ Not just keyword matching
```

### Product Recommendations

**Scenario:** "Find similar products"

```
User views: "Wireless Keyboard"→ Vector search finds similar products→ Based on features, not just category
```

### Academic Research

**Scenario:** Find related research papers

```
Input: Paper abstract→ Vector search finds similar papers→ By research topic/methodology, not just keywords
```

## Architecture

## Installation & Setup

### Install VSS Extension

VSS requires the `HNSW_ENABLE_EXPERIMENTAL_PERSISTENCE` setting when the database is persistent. Initialize the schema and index inside the connection's `init:` block:

flapi.yaml

```
duckdb:  db_path: ./flapi_cache.db  access_mode: READ_WRITE  hnsw_enable_experimental_persistence: trueconnections:  vector-db:    init: |      INSTALL vss;      LOAD vss;      CREATE TABLE IF NOT EXISTS document_embeddings (        id VARCHAR PRIMARY KEY,        content TEXT,        embedding FLOAT[1536],  -- OpenAI ada-002 / text-3-small dimension        metadata JSON,        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP      );      CREATE INDEX IF NOT EXISTS embedding_idx      ON document_embeddings      USING HNSW (embedding);
```

### Or Install Faiss

flapi.yaml

```
connections:  vector-db-faiss:    init: |      INSTALL faiss FROM community;      LOAD faiss;      CREATE TABLE IF NOT EXISTS document_embeddings (        id VARCHAR PRIMARY KEY,        content TEXT,        embedding FLOAT[],        metadata JSON      );
```

Refer to the upstream [Faiss extension docs](https://duckdb.org/community_extensions/extensions/faiss.html) for index creation syntax — flAPI does not add or remove any extension-specific functions.

## Example 1: Knowledge Base Search

### Step 1: Generate and Store Embeddings

This is typically run as a one-time data loader (or background job), not through flAPI:

```
# generate_embeddings.pyimport openaiimport duckdbconn = duckdb.connect('flapi_cache.db')documents = [    {"id": "doc1", "content": "How to reset your password: Go to settings..."},    {"id": "doc2", "content": "API authentication uses JWT tokens..."},    {"id": "doc3", "content": "Database backup runs daily at 2am..."},]for doc in documents:    emb = openai.embeddings.create(        input=doc['content'],        model="text-embedding-3-small",    ).data[0].embedding    conn.execute(        """        INSERT INTO document_embeddings (id, content, embedding, metadata)        VALUES (?, ?, ?, ?)        """,        [doc['id'], doc['content'], emb, '{"category": "docs"}'],    )conn.close()
```

### Step 2: Create Search Endpoint

sqls/semantic\_search.sql

```
WITH query_embedding AS (  -- The client passes the pre-computed embedding as a JSON array string  SELECT CAST('{{{ params.embedding }}}' AS FLOAT[1536]) AS emb)SELECT  d.id,  d.content,  d.metadata,  array_cosine_similarity(d.embedding, q.emb) AS similarityFROM document_embeddings d, query_embedding qWHERE 1=1{{#params.category}}  AND d.metadata->>'category' = '{{{ params.category }}}'{{/params.category}}ORDER BY similarity DESCLIMIT {{#params.limit}}{{ params.limit }}{{/params.limit}}{{^params.limit}}10{{/params.limit}}
```

sqls/semantic\_search.yaml

```
url-path: /search/semantic/method: POSTtemplate-source: semantic_search.sqlconnection:  - vector-dbrequest:  - field-name: embedding    field-in: body    description: Query embedding as JSON array string (e.g. "[0.1,0.2,...]")    required: true    validators:      - type: string        min: 100        # JSON array of 1536 floats is long        max: 200000        preventSqlInjection: false  # JSON contains commas; disable keyword check  - field-name: category    field-in: body    description: Filter by metadata category    required: false    validators:      - type: string        regex: '^[A-Za-z0-9_-]{1,50}$'  - field-name: limit    field-in: body    description: Maximum results    required: false    validators:      - type: int        min: 1        max: 100
```

### Step 3: Search from Client

```
# client.pyimport openaiimport requestsquery = "How do I authenticate with the API?"query_embedding = openai.embeddings.create(    input=query,    model="text-embedding-3-small",).data[0].embeddingresponse = requests.post(    'http://localhost:8080/search/semantic/',    json={        'embedding': str(query_embedding),  # pass as JSON string        'limit': 5,    },)for doc in response.json()['data']:    print(f"{doc['similarity']:.2f}: {doc['content'][:100]}...")
```

## Example 2: Hybrid Search (Vectors + Keywords)

Combine semantic search with traditional filters:

sqls/hybrid\_search.sql

```
WITH query_embedding AS (  SELECT CAST('{{{ params.embedding }}}' AS FLOAT[1536]) AS emb)SELECT  d.id,  d.content,  d.metadata,  array_cosine_similarity(d.embedding, q.emb) AS vector_score,  CASE    WHEN LOWER(d.content) LIKE LOWER('%{{{ params.keywords }}}%') THEN 0.2    ELSE 0  END AS keyword_score,  (array_cosine_similarity(d.embedding, q.emb)   + CASE       WHEN LOWER(d.content) LIKE LOWER('%{{{ params.keywords }}}%') THEN 0.2       ELSE 0     END  ) AS final_scoreFROM document_embeddings d, query_embedding qWHERE 1=1  {{#params.doc_type}}  AND d.metadata->>'type' = '{{{ params.doc_type }}}'  {{/params.doc_type}}  {{#params.date_after}}  AND d.created_at >= DATE '{{{ params.date_after }}}'  {{/params.date_after}}ORDER BY final_score DESCLIMIT 20
```

**Benefits:**

*   Semantic understanding (vectors)
*   Exact phrase matching (keywords)
*   Metadata filtering (structured data)
*   Date ranges, categories, tags

## Example 3: RAG for AI Agents (MCP)

Expose vector search as an MCP tool for Claude/GPT by adding an `mcp-tool:` section:

sqls/rag\_search.yaml

```
url-path: /rag/search/method: POSTmcp-tool:  name: search_knowledge_base  description: |    Search the company knowledge base semantically. Use this to find relevant    documentation, policies, or information to answer user questions.  result-mime-type: application/jsontemplate-source: semantic_search.sqlconnection:  - vector-dbrequest:  - field-name: embedding    field-in: body    description: Query embedding as JSON array string    required: true    validators:      - type: string        min: 100        max: 200000        preventSqlInjection: false  - field-name: category    field-in: body    required: false    validators:      - type: string        regex: '^[A-Za-z0-9_-]{1,50}$'  - field-name: limit    field-in: body    required: false    validators:      - type: int        min: 1        max: 100
```

## Performance Optimization

### Index Selection

**HNSW (Hierarchical Navigable Small World):**

```
-- Best for: < 10M vectors, balanced speed/accuracyCREATE INDEX embedding_idx ON document_embeddingsUSING HNSW (embedding)WITH (M = 16, ef_construction = 200);
```

See the [VSS extension docs](https://duckdb.org/docs/stable/core_extensions/vss.html) for the full list of HNSW parameters.

**Faiss IVF / quantized indexes** are created via the Faiss extension's table/index functions — refer to the upstream extension docs.

### Query Optimization

```
-- Slow: full table scanSELECT * FROM document_embeddingsORDER BY array_cosine_similarity(embedding, query_emb) DESCLIMIT 10;-- Fast: HNSW index + LIMIT (DuckDB pushes down to the index)SELECT * FROM document_embeddingsORDER BY array_cosine_similarity(embedding, query_emb) DESCLIMIT 10;
```

### Caching Popular Queries

```
# Cache pre-computed embeddings for popular documents in DuckLakecache:  enabled: true  table: popular_docs_cache  schedule: 6h  template-file: popular_docs_cache.sql
```

sqls/popular\_docs\_cache.sql

```
INSERT INTO {{cache.catalog}}.{{cache.schema}}.{{cache.table}}SELECT *FROM document_embeddingsWHERE id IN (  SELECT document_id  FROM search_analytics  GROUP BY document_id  HAVING COUNT(*) > 100)
```

## Embedding Models Comparison

| Model | Dimension | Cost | Speed | Quality |
| --- | --- | --- | --- | --- |
| **OpenAI text-embedding-3-small** | 1536 | $0.00002/1K | Fast | Good |
| **OpenAI text-embedding-3-large** | 3072 | $0.00013/1K | Medium | Best |
| **OpenAI ada-002 (legacy)** | 1536 | $0.0001/1K | Fast | Excellent |
| **Cohere embed-v3** | 1024 | $0.0001/1K | Fast | Excellent |
| **Sentence Transformers** | 384-768 | Free | Fast | Good |

**Recommendation:** Start with **OpenAI text-embedding-3-small** (cheapest, good quality).

## Distance Metrics

### Cosine Similarity (most common)

```
array_cosine_similarity(embedding1, embedding2)-- Returns: 0 to 1 (higher = more similar)
```

**Use when:** embeddings are normalized (like OpenAI).

### Euclidean Distance

```
array_distance(embedding1, embedding2)-- Returns: 0 to infinity (lower = more similar)
```

**Use when:** magnitude matters.

### Dot Product

```
array_inner_product(embedding1, embedding2)-- Returns: -infinity to infinity (higher = more similar)
```

**Use when:** faster than cosine, embeddings normalized.

## Security & Best Practices

### 1\. Generate Embeddings Server-Side

Don't trust client-supplied embeddings if the search endpoint represents a security boundary. Generate the embedding from raw query text in a trusted backend, then call flAPI.

### 2\. Rate Limiting

Use flAPI's per-endpoint rate limiting to prevent abuse:

```
rate-limit:  enabled: true  max: 60        # requests  interval: 60   # seconds → 60/min
```

### 3\. Input Validation

```
request:  - field-name: embedding    field-in: body    required: true    validators:      - type: string        min: 100        max: 200000        preventSqlInjection: false
```

### 4\. Content Filtering

Use the `auth` context in templates to restrict results per user:

```
WHERE (d.metadata->>'visibility' = 'public'       OR d.metadata->>'owner_id' = '{{{ auth.username }}}')
```

## Troubleshooting

### Issue: Slow vector searches

**Problem:** Queries take > 100ms

**Solutions:**

1.  Create an HNSW index (if missing)
2.  Reduce vector dimension (use a smaller model)
3.  Use Faiss for large datasets (> 1M vectors)

### Issue: Poor search quality

**Problem:** Irrelevant results returned

**Solutions:**

1.  Use hybrid search (combine with keywords)
2.  Try a better embedding model
3.  Add metadata filters (category, date, etc.)
4.  Re-generate embeddings with a domain-specific model

### Issue: High embedding costs

**Problem:** OpenAI embedding costs too high

**Solutions:**

1.  Use a cheaper model (`text-embedding-3-small`)
2.  Cache embeddings (don't regenerate)
3.  Batch embedding generation (up to 2048 inputs per request)
4.  Consider open-source models (Sentence Transformers)

### Issue: Index creation fails

**Error:** `Index too large` or HNSW memory pressure

```
-- Increase memory or switch to Faiss IVF for very large setsSET memory_limit = '8GB';
```

## Scaling Considerations

### Up to 100K vectors

```
init: |  INSTALL vss;  LOAD vss;  CREATE INDEX embedding_idx ON document_embeddings USING HNSW (embedding);
```

### 100K - 10M vectors

```
init: |  INSTALL faiss FROM community;  LOAD faiss;  -- See Faiss extension docs for IVF index creation
```

### 10M+ vectors

Use Faiss with quantization (e.g. SQ8). Refer to the upstream extension docs for the current syntax.

## Next Steps

*   **[Vector Search RAG Example](/docs/examples/vector-search-rag.md)**: Complete RAG implementation
*   **[AI Integration](/docs/ai-integration/mcp-overview.md)**: MCP tools for AI agents
*   **[Claude Integration](/docs/ai-integration/claude-integration.md)**: Use with Claude Desktop
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Dynamic vector queries
*   **[Caching](/docs/guides/caching/setup.md)**: Cache popular embeddings
*   **[Google Sheets](/docs/guides/connections/google-sheets.md)**: Combine with spreadsheet data

## Additional Resources

*   **[DuckDB VSS Extension](https://duckdb.org/docs/stable/core_extensions/vss.html)**: Official VSS docs
*   **[DuckDB Faiss Extension](https://duckdb.org/community_extensions/extensions/faiss.html)**: Faiss extension docs
*   **[OpenAI Embeddings](https://platform.openai.com/docs/guides/embeddings)**: Embedding generation guide
*   **[Faiss Wiki](https://github.com/facebookresearch/faiss/wiki)**: Faiss documentation

---

**RAG Best Practice:**

For production RAG systems, use **hybrid search** (vectors + keywords + metadata filters). Pure vector search can miss exact matches, while pure keyword search misses semantic meaning. Combining both gives the best results.

([🍪 Cookie Settings](#cookie-settings))

# flapii CLI

`flapii` is the command-line client for the flAPI Configuration Service. It talks to a running `flapi` server over HTTP (`/api/v1/_config/*`) to inspect and manage endpoints, SQL templates, caches, and project settings at runtime — no restart required.

`flapii` is a separate binary from the `flapi` server. For the server's CLI flags (`-c`, `-p`, `--log-level`, `--validate-config`, `--config-service`, `--config-service-token`, `--no-telemetry`) and the `flapi pack` / `info` / `unpack` self-packaging subcommands, see the dedicated [Server CLI](/docs/tools/server-cli.md) reference.

## Installation

`flapi` (server) and `flapii` (CLI) are shipped together. Both binaries live inside the `flapi-io` PyPI wheel and inside the GitHub release archive for your platform.

### Install via PyPI (recommended)

```
pip install flapi-ioflapi  --help        # serverflapii --help        # CLI
```

Both `flapi` and `flapii` entry points are installed on your `PATH` by a single `pip install`. The package name is `flapi-io` because `flapi` is taken on PyPI — there is no separate `flapii` distribution.

### Standalone binaries

Each GitHub release ships pre-compiled archives that contain **both** binaries:

| Asset | Contents |
| --- | --- |
| `flapi-linux-amd64.tar.gz` | `flapi` (server) + `flapii` (CLI) |
| `flapi-linux-arm64.tar.gz` | `flapi` + `flapii` |
| `flapi-macos-arm64.tar.gz` | `flapi` + `flapii` |
| `flapi-windows-amd64.zip` | `flapi.exe` + `flapii.exe` |

Download from the [Releases page](https://github.com/DataZooDE/flapi/releases), extract, and put both binaries on your `PATH`:

```
curl -L https://github.com/DataZooDE/flapi/releases/latest/download/flapi-linux-amd64.tar.gz \  | tar xzchmod +x flapi flapii./flapii --help
```

### Try without installing

```
uvx --from flapi-io flapii --helpuvx --from flapi-io flapi  -c flapi.yaml     # run the server the same way
```

## Talking to a server

`flapii` needs a running `flapi` server with the Configuration Service enabled:

```
# Start the server with the config service API exposed./flapi -c flapi.yaml --config-service --config-service-token "$FLAPI_CONFIG_SERVICE_TOKEN"
```

Point `flapii` at it via flags or environment variables:

```
export FLAPI_BASE_URL=http://localhost:8080export FLAPI_CONFIG_SERVICE_TOKEN="your-token"flapii pingflapii endpoints list
```

## Global Options

The following flags work on every subcommand:

| Flag | Description | Env var |
| --- | --- | --- |
| `-c, --config <path>` | Path to `flapi.yaml` (used to derive defaults like base URL) | —   |
| `-u, --base-url <url>` | Base URL of the `flapi` server | `FLAPI_BASE_URL` |
| `--auth-token <token>` | Bearer token for endpoint authentication | `FLAPI_TOKEN` |
| `-t, --config-service-token <token>` | Token for the Configuration Service API | `FLAPI_CONFIG_SERVICE_TOKEN` |
| `--timeout <seconds>` | HTTP request timeout | `FLAPI_TIMEOUT` |
| `--retries <count>` | Number of retries for failed requests | `FLAPI_RETRIES` |
| `--insecure` | Disable TLS certificate verification | —   |
| `-o, --output <format>` | Output format: `json` (default) or `table` | —   |
| `--json-style <style>` | JSON key casing: `camel` (default) or `hyphen` | —   |
| `--debug-http` | Print HTTP request and response bodies | —   |
| `-q, --quiet` | Suppress non-error output | —   |
| `-y, --yes` | Assume yes for confirmation prompts | —   |

Additional environment variables read by the CLI:

*   `FLAPI_GEMINI_KEY` — Gemini API key used by `flapii project init --ai` and `flapii endpoints wizard --ai`.

## Subcommands

The CLI exposes the following command groups (verify with `flapii --help`):

### `ping`

```
flapii ping
```

Checks that the configured server is reachable and the token (if any) is accepted.

### `config`

Inspect and manage server-side configuration state.

```
flapii config showflapii config validateflapii config log-level getflapii config log-level set debugflapii config log-level list
```

### `project`

```
flapii project get                 # Read project metadata from the serverflapii project init                # Scaffold a new flapi project in the current directoryflapii project init my-api         # Scaffold into a new directoryflapii project init my-api --force --no-examples --advanced --ai
```

`project init` flags:

*   `-f, --force` — overwrite existing files
*   `--skip-validation` — skip post-scaffold config validation
*   `--no-examples` — only create directories and `flapi.yaml`
*   `--advanced` — include extra template files
*   `--ai` — use Gemini-assisted configuration (requires `FLAPI_GEMINI_KEY`)

### `endpoints`

```
flapii endpoints listflapii endpoints get /customersflapii endpoints createflapii endpoints update /customersflapii endpoints delete /customersflapii endpoints validate /customersflapii endpoints reload /customersflapii endpoints wizard
```

`endpoints reload <path>` calls `POST /api/v1/_config/endpoints/{slug}/reload` to apply changes without a server restart.

### `templates`

```
flapii templates listflapii templates get /customersflapii templates update /customersflapii templates expand /customers --params '{"id":"123"}'flapii templates test /customers
```

### `cache`

```
flapii cache listflapii cache get /customersflapii cache update /customersflapii cache template /customersflapii cache update-template /customersflapii cache refresh /customers
```

### `schema`

```
flapii schema getflapii schema refreshflapii schema connectionsflapii schema tables
```

### `mcp`

```
flapii mcp tools listflapii mcp tools get <name>flapii mcp prompts listflapii mcp prompts get <name>flapii mcp resources listflapii mcp resources get <name>
```

## Common Workflows

### Validate an endpoint before reloading

```
flapii endpoints validate /customersflapii endpoints reload /customers
```

### Inspect what SQL a template produces

```
flapii templates expand /customers --params '{"segment":"ENTERPRISE"}'
```

### Refresh a cached endpoint

```
flapii cache refresh /customersflapii cache get /customers --output table
```

### Scaffold and run a new project

```
flapii project init my-first-apicd my-first-api./flapi -c flapi.yaml
```

## CI/CD Integration

```
name: Validate flAPI configurationon: [push, pull_request]jobs:  validate:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v4      - name: Install flAPI        run: |          pip install flapi-io flapii      - name: Validate configuration        run: flapi --validate-config -c flapi.yaml
```

For per-endpoint validation against a running server, use `flapii endpoints validate <path>` in a job that first starts `flapi --config-service` in the background.

## VS Code Integration

A VS Code extension that wraps `flapii` is on the roadmap. See [VS Code Extension](/docs/tools/vscode-extension.md) for current status.

## Server CLI

The `flapi` server binary has its own reference page covering its flags, environment variables, and the `pack` / `info` / `unpack` self-packaging subcommands:

➡ **[Server CLI reference](/docs/tools/server-cli.md)**

That page is the canonical source for everything the `flapi` binary itself accepts; `flapii` (documented above) is the client tool that talks to a running server.

## Troubleshooting

### `flapii: command not found`

Install via `pip install flapii` or download a standalone binary from the [Releases page](https://github.com/DataZooDE/flapi/releases).

### `Error: connection refused`

`flapii` could not reach the server. Check `FLAPI_BASE_URL` (default `http://localhost:8080`) and that the server is running with `--config-service` enabled.

### `401 Unauthorized` from the Configuration Service

Provide the token via `-t/--config-service-token` or `FLAPI_CONFIG_SERVICE_TOKEN`. The server requires either `Authorization: Bearer <token>` or `X-Config-Token: <token>`.

## Next Steps

*   **[Configuration Service](/docs/tools/configuration-service.md)**: Manage configuration at runtime
*   **[VS Code Extension](/docs/tools/vscode-extension.md)**: Roadmap and preview
*   **[Configuration Guide](/docs/getting-started/configuration.md)**: Learn all configuration options
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Master template syntax for `templates expand` and `templates test`
*   **[Build from Source](/docs/getting-started/build-from-source.md)**: Compile flAPI locally

Need help? Join our [GitHub Discussions](https://github.com/DataZooDE/flapi/discussions).

([🍪 Cookie Settings](#cookie-settings))

# Config Service REST API

The Config Service is an authenticated REST API mounted at `/api/v1/_config/*` that lets you inspect and mutate flAPI's runtime configuration — endpoints, templates, caches, schema introspection, log level, and project metadata — without restarting the server.

It is the foundation that the `flapii` CLI and the MCP `flapi_*` administration tools wrap.

## Activation

The Config Service is **disabled by default**. Start the server with `--config-service` to mount the routes:

```
# Auto-generated token printed to stdoutflapi --config-service# Explicit token via flagflapi --config-service --config-service-token "your-secret-token"# Or via environment variableexport FLAPI_CONFIG_SERVICE_TOKEN="your-secret-token"flapi --config-service
```

Without `--config-service`, requests to `/api/v1/_config/*` will not be routed at all.

## Authentication

Every endpoint (except `/health` and `/api/v1/doc.yaml`) requires a bearer token. Two header forms are accepted:

```
Authorization: Bearer <token>
```

```
X-Config-Token: <token>
```

Missing or invalid token returns:

```
HTTP/1.1 401 UnauthorizedContent-Type: text/plainUnauthorized: Invalid or missing token
```

## Base Path

All routes are mounted under `/api/v1/_config/`. The Swagger document is the one exception: `GET /api/v1/doc.yaml`.

## Endpoint Reference

### Project Configuration

#### `GET /api/v1/_config/project`

Retrieve the parsed project configuration.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/project
```

```
{  "project-name": "my-api",  "project-description": "Customer data API",  "server-name": "localhost",  "http-port": 8080,  "template": { "path": "./sqls" }}
```

#### `GET /api/v1/_config/environment-variables`

List the whitelisted environment variables and whether each is currently set.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/environment-variables
```

```
{  "variables": [    { "name": "DB_PASSWORD", "value": "***", "available": true },    { "name": "API_KEY", "value": null, "available": false }  ]}
```

---

### Endpoint Management

Endpoints are identified by **slug**: REST endpoints use `{METHOD}-{path}` (e.g. `GET-customers`), MCP tools use their tool name. See ([Slug format](#slug-format)).

#### `GET /api/v1/_config/endpoints`

List all configured endpoints.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints
```

```
[  {    "url-path": "/customers",    "method": "GET",    "template-source": "customers.sql",    "connection": ["customers-db"]  }]
```

#### `POST /api/v1/_config/endpoints`

Create a new endpoint.

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{    "url-path": "/orders",    "method": "GET",    "template-source": "orders.sql",    "connection": ["orders-db"]  }' \  http://127.0.0.1:8080/api/v1/_config/endpoints
```

```
{ "success": true, "message": "Endpoint created", "slug": "GET-orders" }
```

#### `GET /api/v1/_config/endpoints/{slug}`

Get a single endpoint configuration.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers
```

#### `PUT /api/v1/_config/endpoints/{slug}`

Replace the endpoint configuration.

```
curl -X PUT -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{ "url-path": "/customers", "method": "GET", "template-source": "customers-v2.sql", "connection": ["customers-db"] }' \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers
```

```
{ "success": true, "message": "Endpoint updated" }
```

#### `DELETE /api/v1/_config/endpoints/{slug}`

Delete an endpoint.

```
curl -X DELETE -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers
```

```
{ "success": true, "message": "Endpoint deleted" }
```

#### `POST /api/v1/_config/endpoints/{slug}/validate`

Validate an endpoint YAML body without persisting it.

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: text/plain" \  --data-binary @endpoint.yaml \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/validate
```

```
{ "valid": true, "errors": [], "warnings": [] }
```

#### `POST /api/v1/_config/endpoints/{slug}/reload`

Reload the endpoint definition from disk (useful after editing files directly).

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/reload
```

```
{ "success": true, "message": "Endpoint reloaded from disk" }
```

#### `GET /api/v1/_config/endpoints/{slug}/parameters`

Return parameter metadata for an endpoint.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/parameters
```

```
{  "endpoint": "/customers",  "method": "GET",  "parameters": [    {      "name": "id",      "in": "query",      "required": false,      "validators": [{ "type": "int", "min": 1 }]    }  ]}
```

#### `POST /api/v1/_config/endpoints/by-template`

Find every endpoint (REST or MCP) bound to a given template file.

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{"template_path":"customers.sql"}' \  http://127.0.0.1:8080/api/v1/_config/endpoints/by-template
```

```
{  "template_path": "customers.sql",  "count": 2,  "endpoints": [    { "url_path": "/customers", "method": "GET", "type": "REST" },    { "mcp_name": "get_customers", "type": "MCP_Tool" }  ]}
```

---

### Template Management

#### `GET /api/v1/_config/endpoints/{slug}/template`

Fetch the Mustache+SQL template body.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/template
```

```
{ "template": "SELECT * FROM customers WHERE 1=1 {{#params.id}} AND id = {{ params.id }} {{/params.id}}" }
```

#### `PUT /api/v1/_config/endpoints/{slug}/template`

Replace the template content.

```
curl -X PUT -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{"template":"SELECT * FROM customers WHERE status = '\''active'\''"}' \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/template
```

```
{ "success": true, "message": "Template updated" }
```

#### `POST /api/v1/_config/endpoints/{slug}/template/expand`

Expand the template with sample parameters; useful for debugging without hitting the database.

Query parameters: `include_variables` (boolean), `validate_only` (boolean).

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{"parameters":{"id":"123","status":"active"}}' \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/template/expand
```

```
{ "expanded": "SELECT * FROM customers WHERE 1=1 AND id = 123 AND status = 'active'" }
```

#### `POST /api/v1/_config/endpoints/{slug}/template/test`

Execute the template against the live connection with sample parameters, capped at 10 rows.

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{"parameters":{"id":"123"}}' \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/template/test
```

```
{  "success": true,  "columns": ["id", "name", "email"],  "rows": [ { "id": 123, "name": "John Doe", "email": "john@example.com" } ]}
```

---

### Cache Management

#### `GET /api/v1/_config/endpoints/{slug}/cache`

Return cache configuration for an endpoint.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/cache
```

```
{  "enabled": true,  "table": "customers_cache",  "schema": "cache",  "schedule": "5m",  "primary-key": ["id"],  "cursor": { "column": "updated_at", "type": "timestamp" }}
```

#### `PUT /api/v1/_config/endpoints/{slug}/cache`

Replace cache configuration.

```
curl -X PUT -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{"enabled":true,"table":"customers_cache","schedule":"10m"}' \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/cache
```

```
{ "success": true, "message": "Cache configuration updated" }
```

#### `GET /api/v1/_config/endpoints/{slug}/cache/template`

Get the cache populate SQL template.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/cache/template
```

```
{ "template": "INSERT INTO {{cache.table}} SELECT * FROM source" }
```

#### `PUT /api/v1/_config/endpoints/{slug}/cache/template`

Update the cache populate template.

```
curl -X PUT -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{"template":"REPLACE INTO {{cache.table}} SELECT * FROM customers"}' \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/cache/template
```

#### `POST /api/v1/_config/endpoints/{slug}/cache/refresh`

Trigger an out-of-band cache refresh.

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/cache/refresh
```

```
{ "success": true, "message": "Cache refresh triggered" }
```

#### `POST /api/v1/_config/endpoints/{slug}/cache/gc`

Run garbage collection on cached snapshots according to the retention policy.

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/endpoints/GET-customers/cache/gc
```

---

### Schema Introspection

#### `GET /api/v1/_config/schema`

Retrieve database schema metadata.

Query parameters: `tables` (boolean), `connections` (boolean), `format=completion`, `connection={name}`.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/schema
```

```
{  "main": {    "tables": {      "customers": {        "is_view": false,        "columns": {          "id":    { "type": "INTEGER", "nullable": false },          "name":  { "type": "VARCHAR", "nullable": false },          "email": { "type": "VARCHAR", "nullable": true }        }      }    }  }}
```

#### `POST /api/v1/_config/schema/refresh`

Force a refresh of the cached schema introspection data.

```
curl -X POST -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/schema/refresh
```

```
{ "success": true, "message": "Schema refreshed" }
```

---

### Server Administration

#### `GET /api/v1/_config/log-level`

Get the current server log level.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/log-level
```

```
{ "level": "info" }
```

#### `PUT /api/v1/_config/log-level`

Change the log level at runtime. Valid values: `debug`, `info`, `warning`, `error`.

```
curl -X PUT -H "Authorization: Bearer $TOKEN" \  -H "Content-Type: application/json" \  -d '{"level":"debug"}' \  http://127.0.0.1:8080/api/v1/_config/log-level
```

```
{ "level": "debug", "message": "Log level updated successfully" }
```

---

### Filesystem Operations

#### `GET /api/v1/_config/filesystem`

Return the project's on-disk structure — useful for tooling that wants to map files to endpoints.

```
curl -H "Authorization: Bearer $TOKEN" \  http://127.0.0.1:8080/api/v1/_config/filesystem
```

```
{  "base_path": "/home/user/project",  "config_file": "flapi.yaml",  "config_file_exists": true,  "template_path": "/home/user/project/sqls",  "tree": [    {      "name": "customers.yaml",      "type": "file",      "path": "customers.yaml",      "extension": ".yaml",      "yaml_type": "endpoint",      "url_path": "/customers"    },    { "name": "customers.sql", "type": "file", "path": "customers.sql", "extension": ".sql" }  ]}
```

#### `GET /api/v1/doc.yaml`

Returns the live OpenAPI 3 specification for the public REST endpoints (user-defined endpoints, not the Config Service itself). **No authentication required.**

```
curl http://127.0.0.1:8080/api/v1/doc.yaml
```

---

## HTTP Status Codes

| Code | Meaning |
| --- | --- |
| 200 | OK — request successful |
| 201 | Created — resource created (e.g. new endpoint) |
| 400 | Bad Request — invalid input, malformed JSON, missing field |
| 401 | Unauthorized — invalid or missing token |
| 404 | Not Found — slug doesn't match any endpoint |
| 500 | Internal Server Error |
| 501 | Not Implemented |
| 503 | Service Unavailable — backend dependency unreachable |

### Error Envelope

Most failures return a JSON body:

```
{  "error": "Validation failed",  "details": ["field-name is required"]}
```

`401 Unauthorized` is the one exception — it returns plain text (`Unauthorized: Invalid or missing token`) so token leakage in JSON logs is avoided.

## Slug Format

| Endpoint type | Format | Example |
| --- | --- | --- |
| REST | `{METHOD}-{path}` | `GET-customers`, `POST-orders` |
| MCP Tool | `{tool_name}` | `get_customers` |
| MCP Resource | `resource-{name}` | `resource-schema` |
| MCP Prompt | `prompt-{name}` | `prompt-analysis` |

Path conversion rules for REST slugs:

*   Leading `/` is removed
*   `/` becomes `-slash-`
*   Other special characters become `-`
*   Empty path becomes `empty`

Examples: `/customers` → `GET-customers`, `/api/v1/users` → `GET-api-slash-v1-slash-users`.

## Related Tooling

The Config Service is the source of truth that higher-level tools wrap:

*   **[flapii CLI](/docs/tools/cli-overview.md)** — wraps every endpoint above as a friendly subcommand (`flapii endpoints list`, `flapii templates expand`, `flapii cache refresh`, etc.).
*   **MCP `flapi_*` tools** — the AI integration surface exposes these same operations as MCP tools so that agents can manage flAPI through the [MCP protocol](/docs/ai-integration/mcp-overview.md).
*   **[Cloud Storage and VFS](/docs/guides/cloud-storage.md)** — pair this API with remote `flapi.yaml` to roll out config changes without touching the filesystem.

([🍪 Cookie Settings](#cookie-settings))

# Configuration Service

flAPI's configuration story has two layers:

1.  **The `flapi.yaml` file** — declarative configuration loaded at server start. Supports environment variable substitution, hierarchical YAML, and modular includes.
2.  **The Configuration Service API** — an optional HTTP API (`/api/v1/_config/*`) the server exposes when started with `--config-service`. The `flapii` CLI is a thin client over this API and lets you inspect and modify configuration without restarting the server.

This page covers both layers.

## Configuration Architecture

## Hierarchical Configuration

### Main Configuration File

`flapi.yaml` is the entry point. Canonical keys use kebab-case (`project-name`, `project-description`), with a few snake\_case keys for DuckDB pass-through settings.

```
# flapi.yamlproject-name: my-api-projectproject-description: Production API for customer data# Template configurationtemplate:  path: './sqls'  environment-whitelist:    - '^FLAPI_.*'    - '^DB_.*'    - '^GOOGLE_.*'# Include external configuration filesconnections: {{include from 'config/connections.yaml'}}# DuckDB engine settings (snake_case keys pass through to DuckDB)duckdb:  db_path: ./cache/flapi.db  access_mode: READ_WRITE  threads: 8  max_memory: 8GB
```

### Modular Configuration with Includes

Break large configurations into modules:

**`config/connections.yaml`:**

```
bigquery-warehouse:  init: |    INSTALL 'bigquery';    LOAD 'bigquery';  properties:    project_id: '${GOOGLE_CLOUD_PROJECT}'    credentials_path: '${GOOGLE_APPLICATION_CREDENTIALS}'postgres-db:  init: |    INSTALL postgres;    LOAD postgres;  properties:    host: '${DB_HOST}'    port: 5432    database: '${DB_NAME}'    username: '${DB_USER}'    password: '${DB_PASSWORD}'customers-parquet:  properties:    path: './data/customers.parquet'
```

## Environment Variables

### `${VAR}` substitution in YAML

flAPI substitutes `${VAR_NAME}` references at load time. A default value can follow a `|`:

```
connections:  production-db:    properties:      host: '${DB_HOST}'      port: ${DB_PORT|5432}      username: '${DB_USER}'      password: '${DB_PASSWORD}'      ssl_mode: '${DB_SSL_MODE|require}'
```

Inside SQL templates (Mustache), use `{{env.VAR_NAME}}` to read whitelisted environment variables.

Only the two interpolation forms above are supported. There is no built-in `${vault:...}` or `${aws-secrets:...}` resolver — wire those secrets in via environment variables sourced from your deployment platform.

### Environment Variable Whitelist

Control which variables are visible to templates:

```
template:  environment-whitelist:    - '^FLAPI_.*'    - '^DB_.*'    - '^GOOGLE_.*'    - '^AWS_.*'    - 'JWT_SECRET'
```

Only whitelisted variables are accessible from `{{env.VAR_NAME}}` in templates. This prevents accidental exposure of unrelated environment variables.

### Environment-aware includes

Vary the configuration by environment by interpolating into the include path:

```
environment: '${FLAPI_ENV|development}'connections: {{include from 'config/connections-${FLAPI_ENV}.yaml'}}
```

Then maintain `config/connections-development.yaml`, `config/connections-staging.yaml`, etc.

## Configuration Validation

### From the command line (server)

The server can validate and exit without starting:

```
./flapi --validate-config -c flapi.yamlecho $?   # 0 = valid, 1 = invalid
```

Use this in CI to fail builds on broken configuration.

### From the command line (CLI client)

When the server is already running with `--config-service`, you can validate the active configuration via the CLI:

```
flapii config validate
```

### Common Validation Errors

**Missing Environment Variable:**

```
Error: Environment variable 'DB_PASSWORD' is required but not setFix: export DB_PASSWORD='your-password'
```

**Invalid YAML Syntax:**

```
Error: YAML parse error on line 23  Expected ':' after key, got '-'Fix: Check indentation and YAML syntax
```

**Missing Include File:**

```
Error: Include file not found: config/missing.yamlFix: Create the file or fix the include path
```

## Configuration Reference (abbreviated)

For the full set of keys, see [`docs/CONFIG_REFERENCE.md`](https://github.com/DataZooDE/flapi/blob/main/docs/CONFIG_REFERENCE.md). Selected highlights:

```
# Project metadataproject-name: my-api-projectproject-description: API for business intelligence# SQL template roottemplate:  path: './sqls'  environment-whitelist:    - '^FLAPI_.*'    - '^DB_.*'# DuckDB engine (snake_case pass-through)duckdb:  db_path: ./cache/flapi.db  access_mode: READ_WRITE  threads: 8  max_memory: 8GB# Data connectionsconnections:  bigquery-warehouse:    init: |      INSTALL 'bigquery';      LOAD 'bigquery';    properties:      project_id: '${GOOGLE_CLOUD_PROJECT}'      credentials_path: '${GOOGLE_APPLICATION_CREDENTIALS}'  postgres-db:    init: |      INSTALL postgres;      LOAD postgres;    properties:      host: '${DB_HOST}'      port: 5432      database: '${DB_NAME}'      username: '${DB_USER}'      password: '${DB_PASSWORD}'# Server (selected keys)server:  http-port: 8080  http-host: 0.0.0.0  log_level: info                  # debug, info, warning, error# Authenticationauth:  default_required: true  jwt_secret: '${JWT_SECRET}'  allowed_roles: [admin, user]# Global rate limitingrate_limit:  enabled: true  max: 100        # requests per interval  interval: 60    # seconds
```

## The Configuration Service API

The Configuration Service exposes runtime endpoints under `/api/v1/_config/*` for listing, inspecting, mutating, and reloading endpoints, templates, and caches.

### Enable it

The API is **off by default**. Enable it on the server:

```
./flapi -c flapi.yaml \        --config-service \        --config-service-token "$FLAPI_CONFIG_SERVICE_TOKEN"
```

*   If `--config-service` is set without a token, the server generates a random one at startup and logs it.
*   The token can also be supplied via the `FLAPI_CONFIG_SERVICE_TOKEN` environment variable.

### Authentication

Every request to `/api/v1/_config/*` must present the token in one of:

```
Authorization: Bearer <token>
```

or

```
X-Config-Token: <token>
```

### Reload an endpoint without restart

The dedicated reload endpoint applies updated YAML or template changes:

```
POST /api/v1/_config/endpoints/{slug}/reload
```

This is what `flapii endpoints reload <path>` calls under the hood. flAPI does not poll the filesystem; reloads are explicit and gated by authentication.

For the full API surface (endpoints, templates, caches, schema, MCP, ping), see [`docs/CONFIG_SERVICE_API_REFERENCE.md`](https://github.com/DataZooDE/flapi/blob/main/docs/CONFIG_SERVICE_API_REFERENCE.md) and the [flapii CLI overview](/docs/tools/cli-overview.md).

## Scaffolding a project

Use the CLI to scaffold a working `flapi.yaml` plus directory layout:

```
flapii project init my-first-api
```

Supported flags:

*   `-f, --force` — overwrite existing files
*   `--skip-validation` — skip post-scaffold validation
*   `--no-examples` — only create directories and `flapi.yaml`
*   `--advanced` — include additional template files
*   `--ai` — use Gemini-assisted setup (requires `FLAPI_GEMINI_KEY`)

There is no built-in `--template <preset>` switch. Pick a starter from the [examples directory](https://github.com/DataZooDE/flapi/tree/main/examples) and copy the relevant `flapi.yaml` if you need a specific data-source preset.

## Best Practices

### Use includes for organization

```
# Good: modular and maintainableconnections: {{include from 'config/connections.yaml'}}auth: {{include from 'config/auth.yaml'}}# Bad: everything in one 1000-line file
```

### Provide defaults for optional values

```
# Good: fallback when env var is unsetport: ${DB_PORT|5432}max_memory: ${FLAPI_MAX_MEMORY|8GB}# Bad: hard failure when env var is missingport: ${DB_PORT}
```

### Keep secrets out of YAML

```
# Good: secrets via environmentpassword: '${DB_PASSWORD}'# Bad: hardcoded secret in version controlpassword: 'my-secret-password'
```

### Validate before deploying

```
# Fast check, no server start./flapi --validate-config -c flapi.yaml# Same against a running serverflapii config validate
```

## Troubleshooting

### Configuration not loading

```
Error: Failed to load configuration file
```

1.  `ls -la flapi.yaml` — does the file exist where the server runs?
2.  `./flapi --validate-config -c flapi.yaml` — exact parser error.
3.  Check the working directory used by your process supervisor.

### Environment variable not resolved

```
Error: Environment variable 'DB_HOST' not found
```

1.  `echo $DB_HOST` in the same shell that launches `flapi`.
2.  Confirm the variable matches the `environment-whitelist` if it is referenced from a template.

### `401 Unauthorized` from the Configuration Service

The server received a request to `/api/v1/_config/*` without a valid token. Set `FLAPI_CONFIG_SERVICE_TOKEN` for both the server and `flapii`, or pass `-t/--config-service-token` explicitly.

## Next Steps

*   **[Configuration Guide](/docs/getting-started/configuration.md)**: Complete configuration reference
*   **[CLI Overview](/docs/tools/cli-overview.md)**: `flapii` and the `flapi` server CLI
*   **[VS Code Extension](/docs/tools/vscode-extension.md)**: Visual config editing (preview)
*   **[Endpoints Overview](/docs/endpoints/overview.md)**: Configure API endpoints
*   **[Connections Overview](/docs/guides/connections/overview.md)**: Set up data sources
*   **[Deployment Guide](/docs/getting-started/deployment.md)**: Production deployment
*   **[Authentication](/docs/endpoints/authentication.md)**: Secure your configuration

([🍪 Cookie Settings](#cookie-settings))

# Server CLI (`flapi`)

`flapi` is the server binary. It loads a configuration file, opens an embedded DuckDB instance, and serves the REST API + MCP endpoints described by your endpoint YAMLs.

The same binary is also the **packager**: `flapi pack` folds an entire config tree (YAML + SQL templates + small data files) into a new self-contained executable, suitable for `scp`\-style deployment.

For the upstream reference document, see [`docs/CLI_REFERENCE.md`](https://github.com/DataZooDE/flapi/blob/main/docs/CLI_REFERENCE.md) on the flAPI repository. For the _client_ CLI (`flapii`, which talks to a running server's configuration API), see [flapii CLI](/docs/tools/cli-overview.md).

## Flags

| Flag | Description | Default |
| --- | --- | --- |
| `-c, --config <path>` | Path to the `flapi.yaml` configuration file | `flapi.yaml` |
| `-p, --port <int>` | Port for the web server (overrides `server.http-port`) | from config |
| `--log-level <level>` | One of `debug`, `info`, `warning`, `error` | `info` |
| `--validate-config` | Validate the configuration file and exit | off |
| `--config-service` | Enable the Configuration Service API (`/api/v1/_config/*`) | off |
| `--config-service-token <token>` | Bearer token required by the Configuration Service | auto-generated when missing |
| `--no-telemetry` | Disable startup/shutdown telemetry events | off |

## Environment variables

The server reads several environment variables. The general precedence rule is **CLI flag > environment variable > built-in default**, so any of these may also be passed via the matching CLI flag.

| Variable | Purpose |
| --- | --- |
| `FLAPI_CONFIG` | Fallback for `-c` / `--config`. Lets you point at `flapi.yaml` purely via the environment. |
| `FLAPI_LOG_LEVEL` | Fallback for `--log-level`. Invalid values exit non-zero with a single-line error — typos like `FLAPI_LOG_LEVEL=DEBUG` (uppercase) surface immediately rather than silently defaulting to `info`. |
| `FLAPI_CONFIG_SERVICE_TOKEN` | Fallback for `--config-service-token`. |
| `FLAPI_NO_TELEMETRY` | Set to `1`, `true`, or `yes` to disable telemetry (same as `--no-telemetry`). |
| `SOURCE_DATE_EPOCH` | Mtime stamped on every entry by `flapi pack`. Set to a fixed value to make `pack` output bit-identical across runs (see ([reproducible builds](#reproducible-builds))). |
| `CODESIGN_IDENTITY` | macOS only. Identity passed to `codesign --sign` after `flapi pack`. Defaults to `-` (ad-hoc). |

Beyond these, **any environment variable whose name matches a regex in `template.environment-whitelist`** may be substituted inside `flapi.yaml` and SQL templates via `{{env.VAR_NAME}}`. That is how secrets like `DB_PASSWORD` reach the running server — they live in the deployment environment, never in the config tree itself.

## Self-packaging subcommands

The `flapi` binary can fold an entire config tree into itself, producing one self-contained executable deployable via `scp`. Three subcommands implement the round-trip:

| Subcommand | What it does |
| --- | --- |
| `flapi pack --in <dir> --out <bin> [--allow-secrets] [--macos-append]` | Bundle `<dir>` into a new self-contained binary at `<bin>` |
| `flapi info` | Print EOCD offset, bundle size, and entry list of the _running_ binary (exit 1 + `"Bundle: none"` when unbundled) |
| `flapi unpack --to <dir>` | Dump the running binary's bundle back to a directory |

### `flapi pack`

```
flapi pack --in <config-dir> --out <new-binary> [--allow-secrets] [--macos-append]
```

| Option | Required | Description |
| --- | --- | --- |
| `--in <dir>` | yes | Directory containing `flapi.yaml` and friends. Walked recursively. |
| `--out <path>` | yes | Path for the bundled output binary. Overwritten if it exists. |
| `--allow-secrets` | no  | Bypass the ([default secret deny list](#secret-deny-list)). Testing only — production users must never set this. |
| `--macos-append` | no  | macOS only. Use the legacy trailing-bytes layout instead of the reserved `__FLAPI/__bundle` segment. The result is **not notarisable** — for local debugging only. |

**Re-pack idempotence.** If the host binary already has a trailing bundle, `pack` strips it from the _copy_ (not the running binary) before writing the new one, so repeated invocations don't grow the output. Same goes for the macOS reserved-segment path: the segment is overwritten in place.

### `flapi info`

```
flapi info
```

Prints the EOCD offset, bundle size, and entry list of the running binary. Exits non-zero with `"Bundle: none (filesystem mode)"` when no bundle is present (Linux/Windows binaries built without `pack` having been run; macOS binaries with an empty placeholder segment).

```
$ ./flapi-prod infoBinary: /opt/flapi/flapi-prodBundle offset: 70123456Bundle size:   12534 bytesEntries (17):  flapi.yaml (1024 bytes)  sqls/customers.yaml (412 bytes)  ...
```

### `flapi unpack`

```
flapi unpack --to <dir>
```

Writes every bundle entry to `<dir>` (creating intermediate directories as needed), preserving paths. Useful for diffing a deployed bundle against a development tree or for audit.

### Worked example

```
# 1. Pack a config tree into a new bundled binaryflapi pack --in ./examples --out flapi-prod# 2. Inspect what's bundled./flapi-prod info# 3. (Optional) extract for debugging or audit./flapi-prod unpack --to /tmp/extracted# 4. One-file deployscp flapi-prod user@host:/opt/flapi/ssh user@host '/opt/flapi/flapi-prod'        # serves bundled config from any cwd
```

### Secret deny list

`flapi pack` enforces a default deny list and refuses files whose relative path matches any of:

*   `*.env` at any depth (e.g. `.env`, `config/.env`)
*   `secrets/` segment at any depth (e.g. `secrets/db.token`, `nested/secrets/api.key`)
*   `*.pem` at any depth
*   `*.key` at any depth

Hitting any of these aborts the pack with a non-zero exit and a clear error naming the offending file. The `--allow-secrets` flag exists for testing (it lets the integration test suite verify the deny list itself); production users must never use it.

The rationale: a bundled binary that contains a database password _is itself a secret_. The whole single-artifact value disappears the moment that artifact becomes sensitive. Credentials come from environment variables at runtime — `AWS_*`, `GOOGLE_*`, `AZURE_*`, `FLAPI_CONFIG_SERVICE_TOKEN`, and `{{env.VAR}}` substitution inside whitelisted YAML.

### Reproducible builds

Set `SOURCE_DATE_EPOCH` before `flapi pack` and the output is byte-identical across runs:

```
SOURCE_DATE_EPOCH=1700000000 flapi pack --in config --out aSOURCE_DATE_EPOCH=1700000000 flapi pack --in config --out bsha256sum a b   # identical
```

The archive itself is deterministic by construction (entries sorted, fixed mtime, no tar-block padding), so once you fix the source binary, the output is a pure function of the input tree.

### macOS notarisation

By default `flapi pack` on macOS writes the archive into a reserved `__FLAPI/__bundle` Mach-O segment that was allocated at link time (16 MiB default; knob `FLAPI_RESERVED_BUNDLE_MIB` at CMake configure time). The segment is overwritten in place, then `flapi pack` re-invokes:

```
codesign --force --sign "${CODESIGN_IDENTITY:--}" <out>
```

The result is suitable for `xcrun notarytool submit`. If the archive doesn't fit (e.g. you tried to bundle a 32 MiB data file into a 16 MiB segment), `flapi pack` exits non-zero with an error message mentioning `FLAPI_RESERVED_BUNDLE_MIB` so you know which knob to rebuild flAPI with.

The legacy `--macos-append` flag falls back to the Linux/Windows trailing-bytes layout. `codesign --verify` will fail on that output because the trailing data is outside what codesign considers signable; `flapi pack` warns to stderr and continues. **Use `--macos-append` only for local debugging**; the binary is intentionally not notarisable.

### See also

*   [Deployment § 5️⃣ Self-Packaged Binary](/docs/getting-started/deployment.md#5%EF%B8%8F%E2%83%A3-self-packaged-binary-single-file-deploy) — when to choose this strategy vs. mounted ConfigMaps, custom Docker images, etc.
*   [`docs/CLI_REFERENCE.md §3`](https://github.com/DataZooDE/flapi/blob/main/docs/CLI_REFERENCE.md#3-self-packaging-subcommands) — full upstream reference
*   [`docs/spec/DESIGN_DECISIONS.md §9`](https://github.com/DataZooDE/flapi/blob/main/docs/spec/DESIGN_DECISIONS.md#9-self-packaging-via-appended-zip) — rationale: ZIP append vs. tar/7z, IFileProvider reuse, `embed://` scheme, reserved-segment on macOS

## Examples

```
# Development: debug logging, examples config./flapi -c examples/flapi.yaml --log-level debug# Production-style run./flapi -c /etc/flapi/flapi.yaml --log-level info# Validate config in CI and exit non-zero on failure./flapi --validate-config -c flapi.yaml# Enable the runtime Configuration Service./flapi -c flapi.yaml --config-service --config-service-token "$FLAPI_CONFIG_SERVICE_TOKEN"# 12-factor: configure everything from the environmentexport FLAPI_CONFIG=/etc/flapi/flapi.yamlexport FLAPI_LOG_LEVEL=infoexport FLAPI_CONFIG_SERVICE_TOKEN=..../flapi
```

## Signal handling

*   `SIGINT` (Ctrl+C) triggers a graceful shutdown: stop accepting new connections, drain in-flight requests, close database connections, exit cleanly.
*   Other signals are not handled explicitly.

## Exit codes

| Code | Meaning |
| --- | --- |
| `0` | Success (normal exit, or `--validate-config` passed, or `flapi pack` / `info` / `unpack` succeeded) |
| `1` | General error: invalid arguments, configuration error, startup failure, validation failure, pack refused a secret-looking file, codesign failed on reserved-segment output, or `flapi info` ran on a binary without a bundle |

## Next Steps

*   **[flapii CLI](/docs/tools/cli-overview.md)** — the _client_ CLI that talks to a running server's configuration service
*   **[Configuration Service](/docs/tools/configuration-service.md)** — manage configuration at runtime over HTTP
*   **[Deployment](/docs/getting-started/deployment.md)** — strategies for shipping `flapi` to production, including the [self-packaged single-file path](/docs/getting-started/deployment.md#5%EF%B8%8F%E2%83%A3-self-packaged-binary-single-file-deploy)
*   **[Configuration Reference](/docs/getting-started/configuration.md)** — every option in `flapi.yaml`

([🍪 Cookie Settings](#cookie-settings))

# VS Code Extension for flAPI

**Preview / Roadmap:**

The flAPI VS Code extension is **not yet released**. The repository (`extensions/vscode/`) contains a work-in-progress and the matching `vscode-build` / `vscode-dev` Makefile targets exist in [`flapi`](https://github.com/DataZooDE/flapi), but no packaged `.vsix` is published yet.

The feature list, settings keys, keyboard shortcuts, and snippet names on this page describe **planned** behavior. They will be confirmed (or revised) when the first preview build ships. Until then, use the [`flapii` CLI](/docs/tools/cli-overview.md) for validation, template expansion, and runtime configuration.

The flAPI VS Code extension is intended to wrap the `flapii` CLI to provide an in-editor development experience for `flapi.yaml`, endpoint YAML, and Mustache SQL templates.

## Planned Features

**Planned — not yet implemented:**

The items below describe the target feature set. Capabilities, names, and bindings are likely to change before the first release.

### Syntax Highlighting

*   YAML configuration files
*   SQL templates with Mustache syntax
*   Inline validation errors

### Real-Time Validation

*   Endpoint configuration errors highlighted
*   SQL syntax checking
*   Connection verification
*   Parameter validation

### Quick Actions

*   Validate endpoint (right-click)
*   Expand template with test parameters
*   Run a test query and preview results

### IntelliSense

*   Auto-complete for configuration fields
*   Parameter suggestions
*   Connection name completion
*   Mustache template helpers

## Installation

The extension is not on the Marketplace yet. To follow progress or build from source, check the [`extensions/vscode/`](https://github.com/DataZooDE/flapi/tree/main/extensions/vscode) directory in the flAPI repository and the `vscode-build` / `vscode-dev` Makefile targets:

```
git clone --recurse-submodules https://github.com/DataZooDE/flapi.gitcd flapimake vscode-build
```

Star the [GitHub repository](https://github.com/DataZooDE/flapi) to get notified when the first preview is published.

## Planned Configuration

**Planned — subject to change:**

The settings keys below are placeholders for the upcoming extension and are not yet read by anything.

```
{  "flapi.configPath": "./flapi.yaml",  "flapi.validateOnSave": true,  "flapi.showQueryResults": true,  "flapi.maxResultRows": 100}
```

## Planned Keyboard Shortcuts

**Planned — subject to change:**

The shortcuts below have not been finalized and may change before the first release.

| Action | Shortcut (Mac) | Shortcut (Windows/Linux) |
| --- | --- | --- |
| Validate Endpoint | `Cmd+K Cmd+V` | `Ctrl+K Ctrl+V` |
| Expand Template | `Cmd+K Cmd+E` | `Ctrl+K Ctrl+E` |
| Run Query | `Cmd+K Cmd+R` | `Ctrl+K Ctrl+R` |

## Planned Snippets

**Planned — subject to change:**

Snippet identifiers below are tentative and not yet shipped.

*   `flapi-endpoint`: Scaffold a new endpoint
*   `flapi-cache`: Add cache configuration
*   `flapi-auth`: Add authentication
*   `flapi-validation`: Add parameter validators

## In the meantime

Everything the extension is planned to do is available today through the CLI:

```
flapii endpoints validate /customersflapii templates expand /customers --params '{"id":"123"}'flapii endpoints reload /customers
```

See [flapii CLI](/docs/tools/cli-overview.md) for the full command surface.

## Next Steps

*   **[CLI Overview](/docs/tools/cli-overview.md)**: Learn about the command-line tools
*   **[Configuration Service](/docs/tools/configuration-service.md)**: Advanced configuration management
*   **[Quickstart Guide](/docs/getting-started/quickstart.md)**: Build your first API
*   **[Configuration Guide](/docs/getting-started/configuration.md)**: Learn all options
*   **[SQL Templating](/docs/concepts/sql-templating.md)**: Master Mustache syntax
*   **[Build from Source](/docs/getting-started/build-from-source.md)**: Compile and develop locally
*   **[Contributing](/docs/getting-started/contributing.md)**: Contribute to the extension

Want help setting up your development environment? [Contact us](/services).

([🍪 Cookie Settings](#cookie-settings))