Recipe System Design

Overview

The Recipe System enables composable workflows by chaining multiple actions together with conditional branching, parallel execution, and credit reservation. Recipes allow users to define multi-step file processing pipelines that execute atomically with proper error handling and rollback.

Core Concepts

What is a Recipe?

A Recipe is a directed acyclic graph (DAG) of actions with:

Steps: Individual action invocations with input/output bindings
Conditions: Branch logic based on intermediate results or file metadata
Parallel Branches: Concurrent execution paths that can merge
Credit Reservation: Upfront cost estimation with actual usage tracking

Recipe vs Action

Aspect	Action	Recipe
Scope	Single operation	Multiple operations
Execution	Synchronous or async	Always async
Credits	Immediate deduction	Reserved upfront, committed on completion
Input	Single file set	Single file set, transformed between steps
Output	Single result	Aggregated results from all steps

Architecture

Component Diagram

┌─────────────────────────────────────────────────────────────────┐
│                          Gateway                                │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐  │
│  │   Recipe    │  │   Recipe    │  │      Recipe             │  │
│  │  Registry   │──│   Engine    │──│   Credit Manager        │  │
│  └─────────────┘  └─────────────┘  └─────────────────────────┘  │
│         │               │                     │                 │
│         │               ▼                     │                 │
│         │        ┌─────────────┐              │                 │
│         │        │   Action    │◄─────────────┘                 │
│         │        │  Executor   │                                │
│         │        └─────────────┘                                │
└─────────│───────────────│───────────────────────────────────────┘
          │               │
          ▼               ▼
    ┌──────────┐    ┌──────────┐
    │Firestore │    │ Workers  │
    │(recipes) │    │          │
    └──────────┘    └──────────┘

Data Flow

1. User submits recipe execution request
          │
          ▼
2. Gateway estimates total credits
          │
          ▼
3. Credits reserved (not deducted)
          │
          ▼
4. Recipe engine starts execution
          │
          ▼
5. Each step executes via Action Executor
          │
          ├──► On step success: track actual credits
          │
          └──► On step failure: stop execution, release reservation
          │
          ▼
6. All steps complete
          │
          ▼
7. Commit actual credits, release excess reservation
          │
          ▼
8. Return aggregated results

Recipe Definition Schema

TypeScript Interface

interface Recipe {
  id: string;
  name: string;
  version: string;
  description: string;
  category: string;

  // Recipe structure
  steps: RecipeStep[];

  // Input/output contracts
  inputSchema: JSONSchema;
  outputSchema: JSONSchema;

  // Cost estimation
  creditEstimation: CreditEstimation;

  // Metadata
  createdAt: string;
  updatedAt: string;
  createdBy: string;
  status: "draft" | "active" | "deprecated";
}

interface RecipeStep {
  id: string;
  name: string;
  type: "action" | "condition" | "parallel" | "transform";

  // Action step
  action?: ActionReference;

  // Condition step (if/then/else branching)
  condition?: {
    expression: string; // CEL expression
    then: RecipeStep[];
    else?: RecipeStep[];
  };

  // Parallel step (concurrent execution)
  parallel?: {
    branches: RecipeStep[][];
    waitFor: "all" | "any" | "first_success";
    failureStrategy: "fail_fast" | "continue" | "ignore";
  };

  // Transform step (data manipulation without action call)
  transform?: {
    expression: string; // CEL expression for data transformation
    outputBinding: string;
  };

  // Input/output bindings
  inputBindings?: Record<string, string>; // Maps step inputs to previous outputs
  outputBinding?: string; // Name for this step's output

  // Error handling
  onError?: ErrorHandler;
  retryPolicy?: RetryPolicy;
  timeout?: string; // Duration string, e.g., "5m"
}

interface ActionReference {
  name: string;
  version?: string; // If omitted, uses active version
  parameters?: Record<string, any>;
}

interface ErrorHandler {
  strategy: "fail" | "skip" | "fallback" | "retry";
  fallbackStep?: RecipeStep;
  maxRetries?: number;
  retryDelay?: string;
}

interface RetryPolicy {
  maxAttempts: number;
  initialDelay: string;
  maxDelay: string;
  backoffMultiplier: number;
  retryableErrors?: string[]; // Error codes that trigger retry
}

interface CreditEstimation {
  strategy: "fixed" | "per_file" | "calculated";
  fixedCredits?: number;
  perFileCredits?: number;
  formula?: string; // CEL expression for complex calculations
  maxCredits: number; // Upper bound for reservation
}

JSON Example

{
  "id": "recipe_pdf_to_translated_markdown",
  "name": "pdf-to-translated-markdown",
  "version": "1.0.0",
  "description": "Convert PDF to Markdown and translate to target language",
  "category": "document-processing",

  "steps": [
    {
      "id": "step_1",
      "name": "inspect-file",
      "type": "action",
      "action": {
        "name": "file-inspect",
        "version": "1.0.0"
      },
      "outputBinding": "inspection"
    },
    {
      "id": "step_2",
      "name": "check-page-count",
      "type": "condition",
      "condition": {
        "expression": "inspection.extended.pageCount <= 100",
        "then": [
          {
            "id": "step_2a",
            "name": "convert-pdf",
            "type": "action",
            "action": {
              "name": "pdf-to-markdown",
              "parameters": {
                "preserveFormatting": true
              }
            },
            "outputBinding": "markdown"
          }
        ],
        "else": [
          {
            "id": "step_2b",
            "name": "fail-large-document",
            "type": "transform",
            "transform": {
              "expression": "'Document too large: ' + string(inspection.extended.pageCount) + ' pages'",
              "outputBinding": "error"
            }
          }
        ]
      }
    },
    {
      "id": "step_3",
      "name": "translate",
      "type": "action",
      "action": {
        "name": "translate-text",
        "parameters": {
          "targetLanguage": "{{parameters.targetLanguage}}"
        }
      },
      "inputBindings": {
        "text": "markdown.content"
      },
      "outputBinding": "translated"
    }
  ],

  "inputSchema": {
    "type": "object",
    "required": ["fileHashes", "targetLanguage"],
    "properties": {
      "fileHashes": {
        "type": "array",
        "items": { "type": "string" },
        "minItems": 1,
        "maxItems": 1
      },
      "targetLanguage": {
        "type": "string",
        "enum": ["es", "fr", "de", "ja", "zh"]
      }
    }
  },

  "outputSchema": {
    "type": "object",
    "properties": {
      "translatedFileHash": { "type": "string" },
      "originalPageCount": { "type": "integer" },
      "wordCount": { "type": "integer" }
    }
  },

  "creditEstimation": {
    "strategy": "calculated",
    "formula": "10 + (inspection.extended.pageCount * 5) + (inspection.sizeBytes / 1048576 * 2)",
    "maxCredits": 1000
  }
}

Condition Expressions (CEL)

Recipes use Common Expression Language (CEL) for condition evaluation and data transformation.

Available Variables

Variable	Type	Description
`input`	object	Original recipe input
`parameters`	object	User-provided parameters
`fileHashes`	string[]	Input file hashes
`{stepBinding}`	object	Output from a previous step (by binding name)
`_context`	object	Execution context (accountId, correlationId, etc.)

Example Expressions

// Check file type
inspection.mimeType == "application/pdf"

// Check page count range
inspection.extended.pageCount > 0 && inspection.extended.pageCount <= 50

// Check file size (under 10MB)
inspection.sizeBytes < 10485760

// String operations
inspection.mimeType.startsWith("image/")

// Conditional value
inspection.extended.pageCount > 10 ? "large" : "small"

// List operations
size(input.fileHashes) == 1

// Null checking
has(inspection.extended) && has(inspection.extended.pageCount)

Credit Model for Recipes

Reservation Flow

┌─────────────────────────────────────────────────────────────────┐
│                    Credit Lifecycle                              │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  1. ESTIMATE                                                     │
│     └─► Calculate max possible credits based on formula          │
│                                                                  │
│  2. RESERVE                                                      │
│     └─► Hold max credits (not deducted from balance)             │
│     └─► Create reservation record with TTL                       │
│                                                                  │
│  3. TRACK (per step)                                             │
│     └─► Record actual credits used                               │
│     └─► Update running total                                     │
│                                                                  │
│  4a. COMMIT (on success)                                         │
│      └─► Deduct actual total from balance                        │
│      └─► Release reservation                                     │
│      └─► Refund = reserved - actual                              │
│                                                                  │
│  4b. RELEASE (on failure)                                        │
│      └─► Deduct credits for completed steps only                 │
│      └─► Release remaining reservation                           │
│      └─► Record partial execution                                │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Credit Estimation Strategies

Fixed Credits

Simple flat rate regardless of input:

{
  "strategy": "fixed",
  "fixedCredits": 50
}

Per-File Credits

Scales with number of input files:

{
  "strategy": "per_file",
  "perFileCredits": 25
}

Calculated Credits

CEL expression for complex calculations:

{
  "strategy": "calculated",
  "formula": "base_cost + (page_count * per_page_cost) + (size_mb * per_mb_cost)",
  "maxCredits": 500
}

Failure Policies

Policy	Behavior	Credit Handling
`refund_all`	Full refund on any failure	Reserved → Released
`refund_unused`	Charge for completed steps	Actual used → Deducted, Rest → Released
`no_refund`	Charge full estimate	Reserved → Committed (minus buffer)

Parallel Execution

Branch Types

Wait for All

All branches must complete successfully:

{
  "type": "parallel",
  "parallel": {
    "branches": [
      [
        {
          "action": {
            "name": "resize-image",
            "parameters": { "size": "thumbnail" }
          }
        }
      ],
      [
        {
          "action": {
            "name": "resize-image",
            "parameters": { "size": "medium" }
          }
        }
      ],
      [
        {
          "action": {
            "name": "resize-image",
            "parameters": { "size": "large" }
          }
        }
      ]
    ],
    "waitFor": "all",
    "failureStrategy": "fail_fast"
  }
}

Wait for Any

Continue when first branch completes:

{
  "parallel": {
    "branches": [...],
    "waitFor": "any",
    "failureStrategy": "continue"
  }
}

Wait for First Success

Continue when first branch succeeds (ignore failures):

{
  "parallel": {
    "branches": [...],
    "waitFor": "first_success",
    "failureStrategy": "ignore"
  }
}

Failure Strategies

Strategy	Behavior
`fail_fast`	Cancel other branches on first failure
`continue`	Wait for all branches regardless of failures
`ignore`	Ignore failures, only care about successes

API Endpoints

Recipe Management

POST   /v1/recipes                           Create a new recipe
GET    /v1/recipes                           List recipes (paginated)
GET    /v1/recipes/{id}                      Get recipe by ID
PUT    /v1/recipes/{id}                      Update recipe
DELETE /v1/recipes/{id}                      Delete recipe (soft delete)
POST   /v1/recipes/{id}/validate             Validate recipe without saving

Recipe Execution

POST   /v1/recipes/{id}/execute              Execute a recipe
GET    /v1/recipes/{id}/estimate             Estimate execution cost
GET    /v1/recipe-executions/{executionId}   Get execution status
GET    /v1/recipe-executions                 List executions (paginated)
POST   /v1/recipe-executions/{id}/cancel     Cancel running execution

Request/Response Examples

Create Recipe

POST /v1/recipes
Content-Type: application/json
Authorization: Bearer <api_key>

{
  "name": "image-enhancement-pipeline",
  "version": "1.0.0",
  "description": "Upscale and enhance images",
  "steps": [...],
  "inputSchema": {...},
  "outputSchema": {...},
  "creditEstimation": {...}
}

HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": "recipe_abc123",
  "name": "image-enhancement-pipeline",
  "version": "1.0.0",
  "status": "active",
  "createdAt": "2024-01-15T10:30:00Z"
}

Execute Recipe

POST /v1/recipes/recipe_abc123/execute
Content-Type: application/json
Authorization: Bearer <api_key>

{
  "fileHashes": ["sha256_abc..."],
  "parameters": {
    "upscaleFactor": 4,
    "enhanceType": "photo"
  },
  "async": true
}

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "executionId": "exec_xyz789",
  "status": "running",
  "creditsReserved": 150,
  "estimatedDuration": "PT2M30S",
  "progressUrl": "/v1/recipe-executions/exec_xyz789"
}

Get Execution Status

GET /v1/recipe-executions/exec_xyz789
Authorization: Bearer <api_key>

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": "exec_xyz789",
  "recipeId": "recipe_abc123",
  "status": "running",
  "progress": {
    "currentStep": "step_2",
    "completedSteps": ["step_1"],
    "percentComplete": 40
  },
  "credits": {
    "reserved": 150,
    "used": 45,
    "estimated": 120
  },
  "stepResults": {
    "step_1": {
      "status": "completed",
      "outputBinding": "inspection",
      "duration": "PT1.5S",
      "creditsUsed": 5
    }
  },
  "startedAt": "2024-01-15T10:30:00Z"
}

Error Handling

Error Types

Error Code	Description	Retry?
`RECIPE_NOT_FOUND`	Recipe ID doesn't exist	No
`RECIPE_INVALID`	Recipe validation failed	No
`INSUFFICIENT_CREDITS`	Not enough credits for reservation	No
`STEP_FAILED`	Individual step execution failed	Configurable
`STEP_TIMEOUT`	Step exceeded timeout	Configurable
`CONDITION_ERROR`	CEL expression evaluation failed	No
`ACTION_NOT_FOUND`	Referenced action doesn't exist	No
`EXECUTION_CANCELLED`	User cancelled execution	No
`INTERNAL_ERROR`	Unexpected system error	Yes

Step-Level Error Handling

{
  "id": "step_with_retry",
  "type": "action",
  "action": { "name": "flaky-service" },
  "retryPolicy": {
    "maxAttempts": 3,
    "initialDelay": "1s",
    "maxDelay": "30s",
    "backoffMultiplier": 2.0,
    "retryableErrors": ["TIMEOUT", "SERVICE_UNAVAILABLE"]
  },
  "onError": {
    "strategy": "fallback",
    "fallbackStep": {
      "type": "action",
      "action": { "name": "backup-service" }
    }
  },
  "timeout": "5m"
}

Recipe Metadata for Actions

Actions participating in recipes declare their composition capabilities:

type RecipeMetadata struct {
    // Type declarations for input/output
    InputTypes  []TypeDefinition  `json:"inputTypes"`
    OutputTypes []TypeDefinition  `json:"outputTypes"`

    // Conditions this action can evaluate
    BranchConditions []BranchCondition `json:"branchConditions,omitempty"`

    // Whether action can run in parallel
    Parallelizable bool `json:"parallelizable"`

    // Estimated duration range
    DurationRange DurationRange `json:"durationRange,omitempty"`
}

type TypeDefinition struct {
    Name        string      `json:"name"`
    MimeTypes   []string    `json:"mimeTypes,omitempty"`
    Schema      JSONSchema  `json:"schema,omitempty"`
    Description string      `json:"description"`
}

type BranchCondition struct {
    Name        string `json:"name"`
    Expression  string `json:"expression"`  // CEL expression
    Description string `json:"description"`
}

Example Action with Recipe Metadata

{
  "name": "file-inspect",
  "version": "1.0.0",
  "recipeMetadata": {
    "inputTypes": [
      {
        "name": "any-file",
        "mimeTypes": ["*/*"],
        "description": "Any file type"
      }
    ],
    "outputTypes": [
      {
        "name": "file-metadata",
        "schema": {
          "type": "object",
          "properties": {
            "mimeType": { "type": "string" },
            "sizeBytes": { "type": "integer" },
            "extended": { "type": "object" }
          }
        },
        "description": "File metadata including MIME type and extended info"
      }
    ],
    "branchConditions": [
      {
        "name": "is-pdf",
        "expression": "output.mimeType == 'application/pdf'",
        "description": "Check if file is a PDF"
      },
      {
        "name": "is-image",
        "expression": "output.mimeType.startsWith('image/')",
        "description": "Check if file is an image"
      },
      {
        "name": "is-small",
        "expression": "output.sizeBytes < 10485760",
        "description": "Check if file is under 10MB"
      }
    ],
    "parallelizable": true,
    "durationRange": {
      "min": "100ms",
      "max": "5s"
    }
  }
}

Firestore Schema

Collections

recipes/
  {recipeId}/
    - name: string
    - version: string
    - description: string
    - category: string
    - steps: array<Step>
    - inputSchema: object
    - outputSchema: object
    - creditEstimation: object
    - status: "draft" | "active" | "deprecated"
    - createdAt: timestamp
    - updatedAt: timestamp
    - createdBy: string (accountId)

recipeExecutions/
  {executionId}/
    - recipeId: string
    - accountId: string
    - status: "pending" | "running" | "completed" | "failed" | "cancelled"
    - input: object
    - currentStep: string
    - completedSteps: array<string>
    - stepResults: map<string, StepResult>
    - creditsReserved: number
    - creditsUsed: number
    - reservationId: string
    - startedAt: timestamp
    - completedAt: timestamp
    - error: object (if failed)

Implementation Phases

Phase 1: Foundation

Firestore schema for recipes and executions
Recipe validation logic
CEL expression evaluation
Credit estimation and reservation

Phase 2: Execution Engine

Sequential step execution
Input/output binding resolution
Condition evaluation
Progress tracking and status updates

Phase 3: Advanced Features

Parallel execution
Retry policies
Error handlers and fallbacks
Execution cancellation

Phase 4: API & UI

REST API endpoints
WebSocket progress streaming
Recipe builder UI (optional)
Execution monitoring dashboard

Security Considerations

CEL Expression Sandboxing: CEL expressions run in a sandboxed environment with no access to system resources
Credit Limits: Maximum credit reservation prevents abuse; requires sufficient balance
Execution Quotas: Per-account limits on concurrent recipe executions
Recipe Ownership: Recipes are scoped to accounts; no cross-account access
Action Access: Recipes can only reference actions the account has access to
Timeout Enforcement: All steps have configurable timeouts with hard limits

Monitoring & Observability

Metrics

recipe_executions_total - Total recipe executions by status
recipe_execution_duration_seconds - Execution time histogram
recipe_step_duration_seconds - Per-step execution time
recipe_credits_reserved - Credits reserved per execution
recipe_credits_used - Actual credits consumed

Logging

{
  "level": "info",
  "msg": "recipe step completed",
  "executionId": "exec_xyz789",
  "recipeId": "recipe_abc123",
  "stepId": "step_2",
  "stepName": "convert-pdf",
  "duration": "1.5s",
  "creditsUsed": 15,
  "outputBinding": "markdown"
}

Tracing

Recipe executions create a trace with spans for each step, enabling end-to-end visibility across action calls.

Overview​

Core Concepts​

What is a Recipe?​

Recipe vs Action​

Architecture​

Component Diagram​

Data Flow​

Recipe Definition Schema​

TypeScript Interface​

JSON Example​

Condition Expressions (CEL)​

Available Variables​

Example Expressions​

Credit Model for Recipes​

Reservation Flow​

Credit Estimation Strategies​

Fixed Credits​

Per-File Credits​

Calculated Credits​

Failure Policies​

Parallel Execution​

Branch Types​

Wait for All​

Wait for Any​

Wait for First Success​

Failure Strategies​

API Endpoints​

Recipe Management​

Recipe Execution​

Request/Response Examples​

Create Recipe​

Execute Recipe​

Get Execution Status​

Error Handling​

Error Types​

Step-Level Error Handling​

Recipe Metadata for Actions​

Example Action with Recipe Metadata​

Firestore Schema​

Collections​

Implementation Phases​

Phase 1: Foundation​

Phase 2: Execution Engine​

Phase 3: Advanced Features​

Phase 4: API & UI​

Security Considerations​

Monitoring & Observability​

Metrics​

Logging​

Tracing​