Transform step: deterministic data manipulation (select, redact, filter, format)

[chubes4]

Summary

Add a Transform step type to data-machine-business — a deterministic, local data manipulation step that operates on DataPacket[] between Fetch and Publish steps. No AI, no external API calls, no data leaving the server.

Why This Matters

Currently the only way to manipulate data between fetch and publish is the AI step, which sends data to external LLM APIs. This is:

• Insecure for PII — sending customer names, emails, addresses to OpenAI/Anthropic is a non-starter for business data
• Wasteful — burning API tokens on mechanical field operations (select, filter, redact)
• Non-deterministic — AI might format CSVs differently each run
• Unauditable — you can't prove what transformation was applied

The Transform step solves all four by running structured operations locally in PHP.

Use Case: Monthly Sales Export (inspired by Katie Keith / Barn2)

Step 1: FETCH (woocommerce_orders)
  → DataPackets with order fields including PII

Step 2: TRANSFORM (deterministic)
  operations:
    - remove_fields: [customer_name, customer_email, billing_address]
    - format: csv, filename: "sales-{month}-{year}.csv"

Step 3: PUBLISH (email)
  → Sends anonymized CSV to accountant

Zero customer data leaves the server. Transform runs in PHP, locally, deterministically.

Design: Operation-Based Pipeline

A Transform step config contains an ordered array of operations. Each operation is simple, composable, and auditable:

{
  "operations": [
    {
      "type": "select_fields",
      "fields": ["order_id", "date", "items", "total", "payment_method"]
    },
    {
      "type": "redact_fields",
      "fields": ["customer_email"],
      "strategy": "hash"
    },
    {
      "type": "filter_rows",
      "field": "status",
      "operator": "in",
      "values": ["completed", "refunded"]
    },
    {
      "type": "format",
      "output": "csv",
      "filename": "report-{month}-{year}.csv"
    }
  ]
}

Core Operations

Operation	What It Does	Config
select_fields	Keep only specified fields	fields: string[]
remove_fields	Drop specified fields	fields: string[]
redact_fields	Apply redaction strategy	fields: string[], strategy: "remove"|"hash"|"mask"|"replace"
rename_fields	Rename field keys	mapping: Record<string, string>
filter_rows	Keep rows matching conditions	field, operator (eq/neq/in/gt/lt/contains), values
sort	Order by field	field, direction: "asc"|"desc"
aggregate	Group + summarize	group_by, aggregations: [{field, function: sum/count/avg/min/max}]
format	Convert to output format	output: "csv"|"json"|"html_table", filename
split	Divide into multiple outputs	field, groups: Record<string, values[]>

Implementation Shape

class TransformStep extends AbstractStep {
    use StepTypeRegistrationTrait;

    // Position 15 (between Fetch at 10 and AI at 20)
    
    public function execute(array $data_packets, EngineData $engine): array {
        $config = $this->getConfig();
        $operations = $config['operations'] ?? [];
        
        foreach ($operations as $op) {
            $data_packets = $this->applyOperation($op, $data_packets);
        }
        
        return $data_packets;
    }
}

Each operation type is a simple class implementing TransformOperation::apply(array $packets, array $config): array.

Why data-machine-business (Not Core)

Content sites (blogs, personal sites) don't need deterministic transforms — the AI step handles their needs (rewriting, summarizing, enriching). Transform is essential for:

• WooCommerce stores (PII in orders)
• Membership sites (user data)
• Agencies (client reporting)
• Enterprise (compliance, GDPR)

This is a business feature — it belongs alongside Google Sheets, Slack, Discord, and WooCommerce in the business extension.

Relationship to Other Components

• Depends on: data-machine core (step type registration, DataPacket, EngineData)
• Used by: WooCommerce handlers (anonymize before email), Google Sheets export (reshape data), any PII-sensitive flow
• Complements: AI step (Transform handles mechanical work → AI handles judgment work, receiving only anonymized data)
• Enables: Email publish handler (core, #341) to deliver formatted/anonymized reports