Cloud Storage and VFS

flAPI can load its main configuration file (flapi.yaml), endpoint definitions, and SQL templates directly from cloud object storage or any HTTPS URL. Under the hood it uses DuckDB's Virtual File System (VFS), so anything DuckDB can read, flAPI can read.

Why

Treating configuration as a remote artifact unlocks a few useful patterns:

Serverless and read-only containers — run flAPI on AWS Lambda, Cloud Run, or Container Apps with no mounted volumes; just point --config at a bucket.
GitOps and CI-driven config rollouts — push a YAML change to S3 and roll endpoints without rebuilding the container image.
Multi-environment fan-out — one image, three buckets: s3://configs-dev/, s3://configs-staging/, s3://configs-prod/.
Centralized template library — share sqls/ templates across many flAPI deployments.

Quick Start

One example per supported scheme. Set credentials in the environment first (see Authentication below), then:

# Local file (default)
flapi --config ./flapi.yaml

# HTTPS (no credentials required for public URLs)
flapi --config https://raw.githubusercontent.com/myorg/configs/main/flapi.yaml

# Amazon S3
flapi --config s3://my-bucket/configs/flapi.yaml

# Google Cloud Storage
flapi --config gs://my-bucket/configs/flapi.yaml

# Azure Blob Storage
flapi --config az://my-container/configs/flapi.yaml

Supported URI Schemes

Scheme	Description	Credential source
`file://`	Local filesystem (also any plain path)	Filesystem permissions
`https://`	HTTPS URL (public or with auth headers)	None required
`http://`	HTTP URL — not recommended for production	None required
`s3://`	Amazon S3	`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`
`s3a://`, `s3n://`	S3-compatible (Hadoop-style URIs)	Same as `s3://`
`gs://`	Google Cloud Storage	`GOOGLE_APPLICATION_CREDENTIALS`
`az://`, `abfs://`	Azure Blob Storage	`AZURE_STORAGE_ACCOUNT` + `AZURE_STORAGE_KEY` (or connection string)

Only file:// and https:// are allowed by default. To use any other scheme, declare a connection that installs and configures httpfs (see Path security).

Authentication

Amazon S3

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_REGION=us-east-1

# Optional: temporary credentials from STS / assumed role
export AWS_SESSION_TOKEN=FwoGZXIvYXdzE...

flapi --config s3://my-bucket/configs/flapi.yaml

When flAPI runs on EC2, ECS, EKS, or Lambda it transparently uses the attached IAM role — no environment variables required. For S3-compatible storage (MinIO, LocalStack), set AWS_ENDPOINT_URL=http://localhost:9000.

Google Cloud Storage

# Option 1: service-account key file
export GOOGLE_APPLICATION_CREDENTIALS=/secrets/sa.json

# Option 2: application default credentials from gcloud
gcloud auth application-default login

flapi --config gs://my-bucket/configs/flapi.yaml

On Compute Engine, Cloud Run, or GKE the attached service account is used automatically.

Azure Blob Storage

# Option A: account name + key
export AZURE_STORAGE_ACCOUNT=mystorageaccount
export AZURE_STORAGE_KEY=base64key==

# Option B: full connection string
export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=...;EndpointSuffix=core.windows.net"

flapi --config az://my-container/configs/flapi.yaml

Managed Identity is used automatically on Azure-hosted compute.

Remote Template Paths

Two ways to load SQL templates from cloud storage.

1. Set `template.path` at the project level

Every endpoint's template-source is resolved relative to this base URL.

# flapi.yaml
project-name: cloud-templates-demo

template:
  path: s3://my-bucket/templates/

# endpoint YAML, hosted at s3://my-bucket/templates/customers.yaml
url-path: /customers
method: GET
template-source: customers.sql   # resolves to s3://my-bucket/templates/customers.sql
connection:
  - cloud-data

2. Per-endpoint full URI

A template-source containing a full URI overrides template.path:

url-path: /orders
method: GET
template-source: https://templates.example.com/orders.sql
connection:
  - local-data

You can mix and match: local flapi.yaml, remote templates; or remote flapi.yaml, local templates.

Path Security

flAPI hardens remote loading with three layers of defense:

Path traversal blocking — any path containing .. is rejected before resolution.
URL-encoded traversal detection — encoded variants such as %2e%2e are decoded and re-checked.
Scheme whitelist — only schemes that are explicitly allowed will be opened.

The defaults are deliberately strict:

Scheme	Allowed by default?
`file://` (and plain paths)	yes
`https://`	yes
`http://`, `s3://`, `s3a://`, `s3n://`, `gs://`, `az://`, `abfs://`	no — must be enabled

To enable a cloud scheme, declare a connection whose init: block installs and loads httpfs (DuckDB's cloud filesystem extension). flAPI detects the scheme registration and adds it to the allowlist:

connections:
  cloud-data:
    init: |
      INSTALL httpfs;
      LOAD httpfs;
      SET s3_region='us-east-1';
    properties:
      bucket: my-data-bucket
      base_path: s3://my-data-bucket/data/

Once a scheme is allowed by any connection, the same scheme can also be used by --config and template.path.

Caching of Remote Files

Remote files are cached locally with an LRU cache so that flAPI doesn't hammer S3/GCS/Azure on every reload. Local files are never cached — they are always read fresh.

Configure caching in the top-level storage: block of flapi.yaml:

storage:
  cache:
    enabled: true     # default: true
    ttl: 300          # cache TTL in seconds, default: 300
    max_size: 50MB    # LRU eviction threshold, default: 50MB

  credentials:
    s3:
      type: environment    # environment | secret | instance_profile
      region: us-east-1
    gcs:
      type: environment    # environment | service_account
      key_file: /secrets/gcs.json
    azure:
      type: environment    # environment | connection_string | managed_identity
      account: mystorageaccount

Caches are invalidated automatically when TTL expires or when the server restarts. They can also be cleared via the config service health endpoint.

Health Check

The Config Service exposes a health endpoint that reports VFS state — useful as a Kubernetes readiness probe or for incident triage.

curl http://127.0.0.1:8080/api/v1/_config/health

{
  "status": "healthy",
  "storage": {
    "status": "healthy",
    "backends": [
      {
        "name": "config",
        "path": "s3://my-bucket/configs/flapi.yaml",
        "accessible": true,
        "latency_ms": 45,
        "scheme": "s3"
      },
      {
        "name": "templates",
        "path": "./sqls/",
        "accessible": true,
        "latency_ms": 2,
        "scheme": "local"
      }
    ],
    "total_latency_ms": 47
  },
  "credentials": {
    "s3_configured": true,
    "gcs_configured": false,
    "azure_configured": false
  }
}

This endpoint does not require authentication, even when the Config Service token is set.

Complete Worked Example

A fully cloud-native deployment with flapi.yaml in S3, endpoint YAML alongside it, and templates in the same bucket.

`s3://my-bucket/configs/flapi.yaml`

project-name: Cloud-Native API
project-description: flAPI configuration served from S3

template:
  path: s3://my-bucket/templates/
  environment-whitelist:
    - '^AWS_.*'

storage:
  cache:
    enabled: true
    ttl: 300
    max_size: 50MB

connections:
  cloud-data:
    init: |
      INSTALL httpfs;
      LOAD httpfs;
      SET s3_region='us-east-1';
    properties:
      base_path: s3://my-data-bucket/parquet/

duckdb:
  access_mode: READ_ONLY

`s3://my-bucket/templates/customers.yaml`

url-path: /customers
method: GET
template-source: customers.sql
connection:
  - cloud-data
request:
  - field-name: country
    field-in: query
    required: false
    validators:
      - type: string

`s3://my-bucket/templates/customers.sql`

SELECT id, name, email, country
FROM read_parquet('{{ conn.cloud-data.base_path }}/customers.parquet')
WHERE 1 = 1
{{#params.country}}
  AND country = '{{ params.country }}'
{{/params.country}}
LIMIT 1000

Run

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=us-east-1

flapi --config s3://my-bucket/configs/flapi.yaml

flAPI loads flapi.yaml from S3, discovers customers.yaml under template.path, resolves customers.sql relative to the same prefix, registers the endpoint, and starts serving GET /customers?country=DE — all without any local files.

Troubleshooting

Access Denied — verify AWS_ACCESS_KEY_ID and that the IAM policy grants s3:GetObject on the config and templates prefixes.
File Not Found — double-check the bucket/key. aws s3 ls s3://my-bucket/configs/ is the fastest sanity check.
Slow startup — use regional endpoints; remote configuration loading adds network latency. The LRU cache absorbs subsequent requests.
Scheme not allowed — confirm a connection installs and loads httpfs so that the scheme makes it onto the allowlist.

Configuration Service — file-level YAML reference and validation
Config Service REST API — runtime endpoint management
flapii CLI — local validation tooling

Why​

Quick Start​

Supported URI Schemes​

Authentication​

Amazon S3​

Google Cloud Storage​

Azure Blob Storage​

Remote Template Paths​

1. Set template.path at the project level​

2. Per-endpoint full URI​

Path Security​

Caching of Remote Files​

Health Check​

Complete Worked Example​

s3://my-bucket/configs/flapi.yaml​

s3://my-bucket/templates/customers.yaml​

s3://my-bucket/templates/customers.sql​

Run​

Troubleshooting​

Related​

Why