Configuration Service

flAPI's configuration service provides powerful tools for managing, validating, and organizing your API configurations. It features hierarchical YAML structure, environment variable support, modular includes, and comprehensive validation.

Configuration Architecture

Hierarchical Configuration

Main Configuration File

The flapi.yaml file is the entry point for your configuration:

# flapi.yaml
project_name: my-api-project
project_description: Production API for customer data

# Template configuration
template:
  path: './sqls'
  environment-whitelist:
    - '^FLAPI_.*'
    - '^DB_.*'
    - '^GOOGLE_.*'

# Include external configuration files
connections: {{include from 'config/connections.yaml'}}
security: {{include from 'config/security.yaml'}}

# DuckDB settings
duckdb:
  db_path: ./cache/flapi.db
  access_mode: READ_WRITE
  threads: 8
  max_memory: 8GB

Modular Configuration with Includes

Break down large configurations into manageable modules:

config/connections.yaml:

bigquery-warehouse:
  init: |
    INSTALL 'bigquery';
    LOAD 'bigquery';
  properties:
    project_id: '${GOOGLE_CLOUD_PROJECT}'
    credentials_path: '${GOOGLE_APPLICATION_CREDENTIALS}'

postgres-db:
  init: |
    INSTALL postgres;
    LOAD postgres;
  properties:
    host: '${DB_HOST}'
    port: 5432
    database: '${DB_NAME}'
    username: '${DB_USER}'
    password: '${DB_PASSWORD}'

customers-parquet:
  properties:
    path: './data/customers.parquet'

config/security.yaml:

enforce-https:
  enabled: true
  ssl-cert-file: './ssl/cert.pem'
  ssl-key-file: './ssl/key.pem'

authentication:
  jwt:
    enabled: true
    secret: '${JWT_SECRET}'
    issuer: 'flapi-server'
    audience: 'api-clients'

rate-limiting:
  enabled: true
  global:
    requests: 10000
    window: 60s

Environment Variables

Using Environment Variables

Reference environment variables using ${VAR_NAME} syntax:

connections:
  production-db:
    properties:
      host: '${DB_HOST}'
      port: ${DB_PORT|5432}         # Default value: 5432
      username: '${DB_USER}'
      password: '${DB_PASSWORD}'
      ssl_mode: '${DB_SSL_MODE|require}'

Environment Variable Whitelist

Control which environment variables are accessible:

template:
  environment-whitelist:
    - '^FLAPI_.*'           # All vars starting with FLAPI_
    - '^DB_.*'              # All database vars
    - '^GOOGLE_.*'          # All Google Cloud vars
    - '^AWS_.*'             # All AWS vars
    - 'JWT_SECRET'          # Specific var

Security Note: Only whitelisted variables are accessible in templates. This prevents accidental exposure of sensitive environment variables.

Environment-Specific Configurations

Create different configurations per environment:

# flapi.yaml
environment: '${FLAPI_ENV|development}'

connections: {{include from 'config/connections-${FLAPI_ENV}.yaml'}}

Files:

• config/connections-development.yaml
• config/connections-staging.yaml
• config/connections-production.yaml

Configuration Validation

Validate Configuration

Check your configuration for errors before deployment:

$ flapii config validate

✓ Configuration file found: flapi.yaml
✓ Syntax valid (YAML parsed successfully)
✓ All includes resolved:
  • config/connections.yaml ✓
  • config/security.yaml ✓
✓ Environment variables resolved:
  • DB_HOST: localhost
  • DB_PORT: 5432 (default)
  • DB_USER: api_user
  • JWT_SECRET: ****** (set)
✓ All connections valid:
  • bigquery-warehouse ✓
  • postgres-db ✓
  • customers-parquet ✓
✓ DuckDB configuration valid
✓ 12 endpoints loaded successfully

Configuration is valid and ready to use.

Common Validation Errors

Missing Environment Variable:

Error: Environment variable 'DB_PASSWORD' is required but not set
Fix: export DB_PASSWORD='your-password'

Invalid YAML Syntax:

Error: YAML parse error on line 23
  Expected ':' after key, got '-'
Fix: Check indentation and YAML syntax

Missing Include File:

Error: Include file not found: config/missing.yaml
Fix: Create the file or fix the include path

Circular Include:

Error: Circular include detected:
  flapi.yaml -> config/a.yaml -> config/b.yaml -> config/a.yaml
Fix: Remove the circular reference

Configuration Structure

Complete Configuration Reference

# ═══════════════════════════════════════════════════════════════
# PROJECT SETTINGS
# ═══════════════════════════════════════════════════════════════
project_name: my-api-project
project_description: API for business intelligence
version: 1.0.0
environment: production

# ═══════════════════════════════════════════════════════════════
# TEMPLATE CONFIGURATION
# ═══════════════════════════════════════════════════════════════
template:
  path: './sqls'                    # Directory for SQL templates
  environment-whitelist:            # Allowed environment variables
    - '^FLAPI_.*'
    - '^DB_.*'
  auto-reload: true                 # Auto-reload on file changes (dev only)

# ═══════════════════════════════════════════════════════════════
# DUCKDB CONFIGURATION
# ═══════════════════════════════════════════════════════════════
duckdb:
  db_path: ./cache/flapi.db         # Database file location
  access_mode: READ_WRITE           # READ_WRITE or READ_ONLY
  threads: 8                        # Number of CPU threads
  max_memory: 8GB                   # Maximum memory usage
  default_order: DESC               # Default sort order
  temp_directory: ./cache/temp      # Temp file location

# ═══════════════════════════════════════════════════════════════
# DATA CONNECTIONS
# ═══════════════════════════════════════════════════════════════
connections:
  # BigQuery
  bigquery-warehouse:
    init: |
      INSTALL 'bigquery';
      LOAD 'bigquery';
    properties:
      project_id: '${GOOGLE_CLOUD_PROJECT}'
      credentials_path: '${GOOGLE_APPLICATION_CREDENTIALS}'
  
  # PostgreSQL
  postgres-db:
    init: |
      INSTALL postgres;
      LOAD postgres;
    properties:
      host: '${DB_HOST}'
      port: 5432
      database: '${DB_NAME}'
      username: '${DB_USER}'
      password: '${DB_PASSWORD}'
    pool:
      min_size: 2
      max_size: 10
      idle_timeout: 300
  
  # Parquet files
  customers-parquet:
    properties:
      path: './data/customers.parquet'

# ═══════════════════════════════════════════════════════════════
# SECURITY SETTINGS
# ═══════════════════════════════════════════════════════════════
enforce-https:
  enabled: true
  ssl-cert-file: './ssl/cert.pem'
  ssl-key-file: './ssl/key.pem'

authentication:
  jwt:
    enabled: true
    secret: '${JWT_SECRET}'
    issuer: 'flapi-server'
    audience: 'api-clients'
    expiration: 3600

rate-limiting:
  enabled: true
  global:
    requests: 10000
    window: 60s
  per-endpoint:
    requests: 1000
    window: 60s

# ═══════════════════════════════════════════════════════════════
# SERVER CONFIGURATION
# ═══════════════════════════════════════════════════════════════
server:
  host: '0.0.0.0'
  port: 8080
  workers: 4
  max-connections: 1000
  timeout: 30s

# ═══════════════════════════════════════════════════════════════
# LOGGING & MONITORING
# ═══════════════════════════════════════════════════════════════
logging:
  level: INFO                       # DEBUG, INFO, WARN, ERROR
  format: json                      # json or text
  output: stdout                    # stdout, file, or both

heartbeat:
  enabled: true
  worker-interval: 10               # Seconds between checks

# ═══════════════════════════════════════════════════════════════
# CORS CONFIGURATION
# ═══════════════════════════════════════════════════════════════
cors:
  enabled: true
  origins:
    - 'https://example.com'
    - 'https://app.example.com'
  methods: ['GET', 'POST', 'PUT', 'DELETE']
  headers: ['Content-Type', 'Authorization']
  credentials: true

Configuration Management

Export Resolved Configuration

See your final configuration with all includes and environment variables resolved:

$ flapii config export

# Outputs the complete, resolved configuration
project_name: my-api-project
connections:
  bigquery-warehouse:
    properties:
      project_id: 'my-gcp-project'
      credentials_path: '/path/to/creds.json'
  postgres-db:
    properties:
      host: 'localhost'
      port: 5432
      database: 'mydb'
      username: 'api_user'
      password: '******'  # Secrets masked

Configuration Diff

Compare configurations across environments:

$ flapii config diff production.yaml staging.yaml

Differences found:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Section               Production         Staging
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
server.workers        4                  2
duckdb.max_memory     8GB                4GB
rate-limiting.global  10000/60s          1000/60s
authentication.jwt    enabled            disabled
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Configuration Templates

Generate configuration templates for common scenarios:

# Generate basic configuration
$ flapii config init --template basic

# Generate with BigQuery setup
$ flapii config init --template bigquery

# Generate with PostgreSQL setup
$ flapii config init --template postgres

# Generate complete production template
$ flapii config init --template production

Advanced Features

Dynamic Configuration Loading

Load configuration programmatically:

# Load specific config file
$ flapi -c custom-config.yaml

# Load from environment variable
$ export FLAPI_CONFIG=/path/to/config.yaml
$ flapi

# Load with overrides
$ flapi -c config.yaml --set server.port=9090

Configuration Profiles

Manage multiple configuration profiles:

# flapi.yaml
profiles:
  development:
    duckdb:
      max_memory: 2GB
    logging:
      level: DEBUG
  
  production:
    duckdb:
      max_memory: 16GB
    logging:
      level: WARN
    enforce-https:
      enabled: true

# Run with specific profile
$ flapi --profile production

Configuration Hot Reload

Enable automatic configuration reload in development:

template:
  auto-reload: true  # Reload on file changes
  reload-interval: 5  # Check every 5 seconds

What reloads:

• ✅ Endpoint configurations
• ✅ SQL templates
• ✅ Environment variables
• ❌ Main config (requires restart)
• ❌ Connection definitions (requires restart)

Configuration Secrets

Manage secrets securely:

# Use external secret managers
connections:
  postgres-db:
    properties:
      password: '${vault:secret/db/password}'  # HashiCorp Vault
      # or
      password: '${aws-secrets:prod/db/password}'  # AWS Secrets Manager

Configuration Best Practices

1. Use Includes for Organization

# ✅ Good: Modular and maintainable
connections: {{include from 'config/connections.yaml'}}
security: {{include from 'config/security.yaml'}}
endpoints: {{include from 'config/endpoints.yaml'}}

# ❌ Bad: Everything in one file (1000+ lines)

2. Environment Variable Naming

# ✅ Good: Consistent naming convention
DB_HOST
DB_PORT
DB_NAME
FLAPI_ENV
JWT_SECRET

# ❌ Bad: Inconsistent naming
host
DatabasePort
db-name
ENV
secret_jwt

3. Use Defaults for Optional Values

# ✅ Good: Provides fallback
port: ${DB_PORT|5432}
max_memory: ${FLAPI_MAX_MEMORY|8GB}

# ❌ Bad: No default (fails if not set)
port: ${DB_PORT}

4. Separate Secrets from Configuration

# ✅ Good: Secrets from environment
password: '${DB_PASSWORD}'

# ❌ Bad: Hardcoded secrets
password: 'my-secret-password'

5. Document Your Configuration

# ✅ Good: Comments explain purpose
connections:
  # Production BigQuery warehouse for analytics
  bigquery-warehouse:
    properties:
      project_id: '${GOOGLE_CLOUD_PROJECT}'
      # Service account must have BigQuery Data Viewer role
      credentials_path: '${GOOGLE_APPLICATION_CREDENTIALS}'

# ❌ Bad: No explanation
connections:
  bq:
    properties:
      project_id: '${X}'

6. Validate Before Deployment

# Always validate configuration changes
$ flapii config validate

# Test with specific environment
$ FLAPI_ENV=production flapii config validate

# Check resolved configuration
$ flapii config export > resolved-config.yaml

Configuration Examples

Example 1: Development Setup

# flapi-dev.yaml
project_name: api-dev
environment: development

template:
  path: './sqls'
  auto-reload: true  # Hot reload in dev

duckdb:
  db_path: ./dev-cache.db
  max_memory: 2GB
  threads: 2

connections:
  local-postgres:
    init: |
      INSTALL postgres;
      LOAD postgres;
    properties:
      host: localhost
      port: 5432
      database: dev_db

logging:
  level: DEBUG
  format: text

server:
  port: 3000

Example 2: Production Setup

# flapi-prod.yaml
project_name: api-production
environment: production

template:
  path: './sqls'
  environment-whitelist:
    - '^FLAPI_.*'
    - '^DB_.*'
    - '^GOOGLE_.*'

duckdb:
  db_path: /var/lib/flapi/cache.db
  max_memory: 16GB
  threads: 16

connections: {{include from 'config/prod-connections.yaml'}}

enforce-https:
  enabled: true
  ssl-cert-file: '/etc/ssl/cert.pem'
  ssl-key-file: '/etc/ssl/key.pem'

authentication:
  jwt:
    enabled: true
    secret: '${JWT_SECRET}'

rate-limiting:
  enabled: true
  global:
    requests: 10000
    window: 60s

logging:
  level: WARN
  format: json
  output: file
  file: /var/log/flapi/app.log

heartbeat:
  enabled: true
  worker-interval: 10

server:
  host: '0.0.0.0'
  port: 8080
  workers: 8

Troubleshooting

Configuration Not Loading

Error: Failed to load configuration file

Check:
1. File exists: ls -la flapi.yaml
2. File permissions: chmod 644 flapi.yaml
3. YAML syntax: flapii config validate
4. Path is correct: pwd

Environment Variable Not Resolved

Error: Environment variable 'DB_HOST' not found

Fix:
1. Check variable is set: echo $DB_HOST
2. Check whitelist: grep DB_ flapi.yaml
3. Export variable: export DB_HOST=localhost

Include File Not Found

Error: Include file 'config/missing.yaml' not found

Fix:
1. Create the file
2. Fix the path in include statement
3. Check relative path from main config

Next Steps

• Configuration Guide: Complete configuration reference
• YAML Syntax: Advanced YAML features and includes
• CLI Overview: Command-line tools for validation
• VS Code Extension: Visual config editing
• Endpoints Overview: Configure API endpoints
• Connections Overview: Set up data sources
• Deployment Guide: Production deployment
• Authentication: Secure your configuration