+ | Field | Type | Description | + |---|---|---| + | `METRICRESULTID` | `VARCHAR` | Unique identifier representing a specific calculation per metric, per experiment, per analysis run. | + | `TREATMENT` | `VARCHAR` | The experiment variant (e.g., Control or Treatment) associated with the metric results. | + | `DIMENSIONNAME` | `VARCHAR` | The name of the dimension being analyzed (e.g., country, platform). | + | `DIMENSIONVALUE` | `VARCHAR` | The corresponding value of the analyzed dimension. | + | `ATTRIBUTEDKEYSCOUNT` | `NUMBER` | Count of unique keys (users, sessions, etc.) attributed to this metric result. | + | `REQUESTTIMESTAMP` | `TIMESTAMP_NTZ` | Timestamp when the metric computation request occurred. | + | `MIN` | `FLOAT` | Minimum observed value for the metric. | + | `MAX` | `FLOAT` | Maximum observed value for the metric. | + | `COUNT` | `NUMBER` | Total number of observations included in the metric calculation. | + | `SUM` | `FLOAT` | Sum of all observed metric values. | + | `MEAN` | `FLOAT` | Average (mean) of the metric values. | + | `P50` | `FLOAT` | 50th percentile (median) metric value. | + | `P95` | `FLOAT` | 95th percentile metric value. | + | `P99` | `FLOAT` | 99th percentile metric value. | + | `VARIANCE` | `FLOAT` | Variance of the metric values. | + | `EXCLUDEDUSERCOUNT` | `NUMBER` | Number of users excluded from the analysis (due to filters, SRM, etc.). | + | `ASOFTIMESTAMP` | `TIMESTAMP_NTZ` | Timestamp representing when the result snapshot was written. | + + To create the results table with the correct structure, run the following SQL statement in Snowflake: + + ```sql + CREATE OR REPLACE TABLE
`flag_name`
`treatment`
`impression_time`
`environment`
`rule_id` | • Map `flag_name` values → `experiment_id` (if multiple flags correspond to the same experiment).
• Cast `evaluation_time` to `TIMESTAMP`.
• Deduplicate on `(user_id, experiment_id)` by keeping the earliest exposure. | + +### A/B Test Impression Logs + +| **Example Raw Schema** | **Transformations** | +| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `experiment_id`
`user_id`
`bucket` or `arm`
`impression_time` | • Standardize `bucket` → `treatment`.
• Standardize `impression_time` → `exposure_timestamp`.
• Deduplicate to keep only the first exposure per user per experiment. | + + +### Event Logging Pipelines (Custom Analytics Events) + +| **Example Raw Schema** | **Transformations** | +| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `event_name`
`event_time`
`properties.experiment_id`
`properties.variant`
`properties.user_id` | • Flatten nested fields (`JSON` → explicit columns).
• Filter to only `event_name = 'experiment_exposure'`.
• Standardize column names to match required schema. | + +## Prepare your assignment table + +Follow these best practices for preparing your assignment table in your data warehouse. + +- **De-duplication**: Keep only the earliest exposure per user per experiment. For example: + + ```sql + QUALIFY ROW_NUMBER() OVER ( + PARTITION BY user_id, experiment_id + ORDER BY exposure_timestamp ASC + ) = 1 + ``` + +- **Consistent Variant Labels**: Standardize variant naming (`control`, `treatment`, `variant_1`) across experiments. Avoid null or empty strings; default to `control` if needed. + +- **Timestamps in UTC**: Store all exposure timestamps in UTC for consistent comparisons across regions. + +- **Stable Identifiers**: Use the same user or account key across
+
+
+Test Connection or Run Query Fails
+ +1. Ensure your table or SQL query is valid and accessible with the credentials tied to your warehouse connection. + +1. Check that you have permission to query all referenced schemas/tables. + +1. Verify that the schema and table names are spelled correctly. +
+
+
+No Data Appears in Preview
+ +If you are using a SQL query, try running it directly in your warehouse to confirm output. + +
+
+
+Column Not Detected or Missing
+ +Verify that your source table/query outputs the required columns: unique identifier, timestamp, and treatment. +
+
+
+Incorrect Environment or Traffic Type Mapping
+ +1. Double-check that each warehouse value (e.g., `UK-Prod`) is mapped to the correct Harness environment (e.g., `Production`). +1. If everything should map to one environment or type, consider using the hardcoded value option instead of column mapping. +
+
\ No newline at end of file
diff --git a/docs/feature-management-experimentation/warehouse-native/setup/configure-metrics.md b/docs/feature-management-experimentation/warehouse-native/setup/configure-metrics.md
new file mode 100644
index 00000000000..141ff3bf89d
--- /dev/null
+++ b/docs/feature-management-experimentation/warehouse-native/setup/configure-metrics.md
@@ -0,0 +1,182 @@
+---
+title: Configuring Metric Sources for Warehouse Native Experimentation
+description: Learn how to configure your metric source tables in your data warehouse for Warehouse Native Experimentation.
+sidebar_label: Configure Metric Sources
+sidebar_position: 4
+---
+
+Assignment Source Not Showing in Experiment Setup
+ +1. Make sure you clicked **Save** after configuration. +1. Confirm that the source hasn’t been deleted, disabled, or restricted to owners only. +
+
+
+Test Connection or Run Query Fails
+ +1. Ensure the table or query is valid and accessible with your warehouse connection credentials. +1. Verify that schemas and table names are spelled correctly. +
+
+
+No Data Appears in Preview
+ +1. Confirm the query/table returns rows for the event(s) you expect. +1. If you are using event filtering in SQL, test the query directly in your warehouse. +
+
+
+Missing Columns
+ +Verify that the required fields exist and are returned by your query. +
+
+
+Timestamp Format Issues
+ +Ensure event timestamps are in a supported `TIMESTAMP` or `DATETIME` format. + +If you are using epoch values (e.g., `EVENT_TIMESTAMP_MS`), convert them in your SQL query. +
+
+
+Incorrect Environment/Traffic Type Mapping
+ +1. Check that each warehouse value is mapped to the intended Harness environment or traffic type. +1. Use hardcoded values if everything should map to a single option. +
+
\ No newline at end of file
diff --git a/docs/feature-management-experimentation/warehouse-native/setup/experiments.md b/docs/feature-management-experimentation/warehouse-native/setup/experiments.md
new file mode 100644
index 00000000000..172048e4e5b
--- /dev/null
+++ b/docs/feature-management-experimentation/warehouse-native/setup/experiments.md
@@ -0,0 +1,59 @@
+---
+title: Create an Experiment
+sidebar_label: Create an Experiment
+description: Learn how to create a cloud experiment in Harness FME.
+sidebar_position: 6
+---
+
+Unable to Delete Source
+ +Check which metric definitions are currently using it. Delete or reassign those metrics before removing the source. +`event_name`
`event_time`
`properties (JSON)` | • Flatten properties into columns for key attributes used in metrics (e.g., `properties.amount` → `event_value`, `properties.tier` → `plan_type`).
• Standardize `event_time` → `event_timestamp`.
• Ensure event names are consistent (`purchase` vs `checkout_completed`). | + +### E-commerce Transaction Logs + +| **Example Raw Schema** | **Transformations** | +| --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `user_id`
`order_id`
`order_time`
`order_amount`
`order_status` | • Map `order_time` → `event_timestamp`.
• Set `event_name = 'purchase'`.
• Use `order_amount` as `event_value`.
• Filter only completed/valid orders (`order_status = 'completed'`). | + +### Custom Business Event Tables + +| **Example Raw Schema** | **Transformations** | +| --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `account_id`
`metric_type`
`metric_value`
`created_at` | • Map `account_id` → `key`.
• Map `metric_type` → `event_name`.
• Standardize `metric_value` → `event_value`. | + +## Prepare your metric table + +- **Consistency with Assignment Source**: Use the same key (`user_id`, `account_id`) in both tables. This is critical for joining exposures to outcomes. +- **De-duplication**: Remove duplicate event logs, which is common in streaming pipelines. Define uniqueness as (`user_id`, `event_name`, `event_timestamp`, `value`) when possible. +- **Timestamps in UTC**: Always store `event_timestamp` in UTC. +- **Flatten Properties Early**: JSON blobs are flexible but slow in downstream queries. Extract only the fields needed for metrics (e.g., `amount`, `plan_tier`, `country`). +- **Event Naming Conventions**: Standardize event names across products and teams. Avoid mixing singular and plural (for example: `purchase` vs `purchases`). +- **Partitioning and Indexing**: Partition large tables by `DATE(event_timestamp)`. Cluster or index by `user_id` or `event_name` for efficient joins with Assignment Sources. +- **Value Handling**: Handle nulls carefully—exclude them or treat them as `0`, depending on the metric's intent. Ensure numeric fields (for example, `event_value`) use consistent units (for example, always USD, not mixed currencies). + +## Example prepared table schema + +| **Column** | **Type** | **Notes** | +| ------------------ | -------------------------- | ----------------------------------------- | +| `user_id` | `STRING` | Required | +| `event_timestamp` | `TIMESTAMP` | Required | +| `event_name` | `STRING` | Required | +| `event_value` | `FLOAT` | Required for average and sum metric types | +| `any_custom_field` | `STRING` / `BOOLEAN` / `NUMERIC` | Optional (flattened from properties) | +| `environment_id` | `STRING` | Recommended | +| `traffic_type` | `STRING` | Optional | + +Once your Metric Source tables are prepared and validated, see [Setting Up an Metric Source](/docs/feature-management-experimentation/warehouse-native/setup/) to connect them in Harness FME. \ No newline at end of file diff --git a/docs/feature-management-experimentation/warehouse-native/setup/metrics.md b/docs/feature-management-experimentation/warehouse-native/setup/metrics.md new file mode 100644 index 00000000000..0098a5d66d6 --- /dev/null +++ b/docs/feature-management-experimentation/warehouse-native/setup/metrics.md @@ -0,0 +1,104 @@ +--- +title: Create a Metric for Warehouse Native Experimentation in Harness FME +sidebar_label: Create a Metric Definition +description: Learn how to create a metric, select a metric source, and set calculation logic for Warehouse Native Experimentation.. +sidebar_position: 5 +--- + +
+
+
+Metric Not Appearing in Experiment Setup
+ +Ensure you’ve clicked **Create** and that the metric includes a valid source and event mapping. + +
+
+
+Incorrect Aggregation
+ +Double-check your **Measure As** setting (for example, `Count`, `Average`, `Sum`, or `Percentage`). + +
+
+
+Event Mismatch
+ +Verify that the selected event name exists in the chosen **Metric Source** and is mapped correctly. +
+
diff --git a/docs/feature-management-experimentation/warehouse-native/static/data-flow.png b/docs/feature-management-experimentation/warehouse-native/static/data-flow.png
new file mode 100644
index 00000000000..68bf594dfe4
Binary files /dev/null and b/docs/feature-management-experimentation/warehouse-native/static/data-flow.png differ
diff --git a/docs/feature-management-experimentation/warehouse-native/static/experiment-entry-filter.png b/docs/feature-management-experimentation/warehouse-native/static/experiment-entry-filter.png
new file mode 100644
index 00000000000..b40c8cabbff
Binary files /dev/null and b/docs/feature-management-experimentation/warehouse-native/static/experiment-entry-filter.png differ
diff --git a/docs/feature-management-experimentation/warehouse-native/static/line-chart.png b/docs/feature-management-experimentation/warehouse-native/static/line-chart.png
new file mode 100644
index 00000000000..58a981a0311
Binary files /dev/null and b/docs/feature-management-experimentation/warehouse-native/static/line-chart.png differ
diff --git a/docs/feature-management-experimentation/warehouse-native/static/share-results.png b/docs/feature-management-experimentation/warehouse-native/static/share-results.png
new file mode 100644
index 00000000000..be89b8b6854
Binary files /dev/null and b/docs/feature-management-experimentation/warehouse-native/static/share-results.png differ
diff --git a/docs/feature-management-experimentation/warehouse-native/static/summarize.png b/docs/feature-management-experimentation/warehouse-native/static/summarize.png
new file mode 100644
index 00000000000..116ddfa259f
Binary files /dev/null and b/docs/feature-management-experimentation/warehouse-native/static/summarize.png differ
diff --git a/docs/feature-management-experimentation/warehouse-native/static/view-metrics.png b/docs/feature-management-experimentation/warehouse-native/static/view-metrics.png
new file mode 100644
index 00000000000..cc2851e262e
Binary files /dev/null and b/docs/feature-management-experimentation/warehouse-native/static/view-metrics.png differ
diff --git a/docs/feature-management-experimentation/warehouse-native/static/view-results.png b/docs/feature-management-experimentation/warehouse-native/static/view-results.png
new file mode 100644
index 00000000000..32d2521a1ad
Binary files /dev/null and b/docs/feature-management-experimentation/warehouse-native/static/view-results.png differ
diff --git a/release-notes/feature-management-experimentation.md b/release-notes/feature-management-experimentation.md
index 4fbdc4bb199..03491c903d5 100644
--- a/release-notes/feature-management-experimentation.md
+++ b/release-notes/feature-management-experimentation.md
@@ -1,7 +1,7 @@
---
title: Feature Management & Experimentation release notes
sidebar_label: Feature Management & Experimentation
-date: 2025-10-15T10:00:00
+date: 2025-10-22T10:00:00
tags: ["fme", "feature management experimentation"]
sidebar_position: 11
---
@@ -12,10 +12,29 @@ import HarnessApiData from '../src/components/HarnessApiData/index.tsx';
These release notes describe recent changes to Harness Feature Management & Experimentation (FME).
-#### Last updated: October 15, 2025
+#### Last updated: October 22, 2025
## October 2025
+### [New Feature] Warehouse Native Experimentation in Beta
+----
+#### 2025-10-22
+
+Harness FME now supports **Warehouse Native Experimentation** in beta. Warehouse Native allows you to run experiments directly in your data warehouse using your own assignment and event data. This approach gives you greater flexibility, transparency, and control over experiment analysis, without needing to export or duplicate data outside your analytics environment.
+
+You can use Warehouse Native Experimentation to:
+
+- Run analyses on experiments with data already stored in your warehouse.
+- Leverage FME's statistical engine and additional measurement techniques for improved accuracy and confidence intervals.
+- Integrate with existing assignment and metric tables in your data warehouse.
+
+To request access for the Warehouse Native Experimentation beta experience, contact [Harness Support](/docs/feature-management-experimentation/fme-support).
+
+#### Related documentation
+
+- [Warehouse Native Experimentation](/docs/feature-management-experimentation/warehouse-native)
+- [Warehouse Native Setup](/docs/feature-management-experimentation/warehouse-native/setup)
+
### [New Feature] Harness Proxy
----
#### 2025-10-15
diff --git a/sidebars.ts b/sidebars.ts
index 4aef83be8fa..29806bddfb2 100644
--- a/sidebars.ts
+++ b/sidebars.ts
@@ -1899,7 +1899,7 @@ const sidebars: SidebarsConfig = {
},
{
type: 'category',
- label: 'Experimentation',
+ label: 'Cloud Experimentation',
link: {
type: 'generated-index',
slug: 'feature-management-experimentation/experimentation',
@@ -1912,6 +1912,19 @@ const sidebars: SidebarsConfig = {
},
],
},
+ {
+ type: 'category',
+ label: 'Warehouse Native Experimentation',
+ className: 'sidebar-item-beta',
+ link: {
+ type: 'doc',
+ id: 'feature-management-experimentation/warehouse-native/index',
+ },
+ collapsed: true,
+ items: [
+ { type: 'autogenerated', dirName: 'feature-management-experimentation/warehouse-native' },
+ ],
+ },
{
type: 'category',
label: 'Release Agent',
diff --git a/src/components/Docs/data/featureManagementExperimentationData.ts b/src/components/Docs/data/featureManagementExperimentationData.ts
index 91e4008a559..c13b88a1436 100644
--- a/src/components/Docs/data/featureManagementExperimentationData.ts
+++ b/src/components/Docs/data/featureManagementExperimentationData.ts
@@ -62,12 +62,19 @@ import { MODULES } from "@site/src/constants"
link: "/docs/feature-management-experimentation/release-monitoring",
},
{
- title: "Experimentation",
+ title: "Cloud Experimentation",
module: MODULES.fme,
description:
"Run experiments and analyze results for data-driven development.",
link: "/docs/feature-management-experimentation/experimentation",
},
+ {
+ title: "Warehouse Native Experimentation",
+ module: MODULES.fme,
+ description:
+ "Run experiments and analyze results for data-driven development in your data warehouse.",
+ link: "/docs/feature-management-experimentation/warehouse-native",
+ },
{
title: "Release Agent",
module: MODULES.fme,
diff --git a/src/components/Docs/data/whnIntegrations.js b/src/components/Docs/data/whnIntegrations.js
new file mode 100644
index 00000000000..321f9fb9a99
--- /dev/null
+++ b/src/components/Docs/data/whnIntegrations.js
@@ -0,0 +1,64 @@
+import React from 'react';
+
+export const dataWarehouses = [
+ {
+ name: 'Snowflake',
+ img: '/provider-logos/whn-integrations/snowflake-logo.png',
+ link: '/docs/feature-management-experimentation/warehouse-native/integrations/snowflake',
+ },
+ {
+ name: 'Amazon Redshift',
+ img: '/provider-logos/whn-integrations/redshift-logo.png',
+ link: '/docs/feature-management-experimentation/warehouse-native/integrations/amazon-redshift',
+ },
+];
+
+// Helper to chunk items into rows of 4
+function chunkArray(array, size) {
+ return array.reduce((acc, _, i) =>
+ (i % size ? acc : [...acc, array.slice(i, i + size)]), []);
+}
+
+// Component to render SDK grid
+export function Section({ title, items }) {
+ const rows = chunkArray(items, 4);
+ return (
+ <>
+ Impact Direction Confusion
+ +Confirm that **Increase** is used for positive outcomes (like conversion or revenue) and **Decrease** for negative ones (like latency or error rate). +{title}
+ {rows.map((row, idx) => ( +
+ {row.map(({ name, img, link }) => (
+
+ ))}
+
+ );
+}
+
+export default function dataWarehousesGrid() {
+ return (
+
+
+
+ ))}
+
+ 
+
+
+ {name}
+
+
+
+ );
+}
diff --git a/src/components/ToolTip/tooltips.json b/src/components/ToolTip/tooltips.json
index 68bdbb7ac08..72664c87e4d 100644
--- a/src/components/ToolTip/tooltips.json
+++ b/src/components/ToolTip/tooltips.json
@@ -11,5 +11,16 @@
"tf-commands": {
"init": "OpenTofu/Terraform command used to initialize a configuration. It downloads and configures providers, modules, and other dependencies."
}
+ },
+ "fme": {
+ "warehouse-native": {
+ "metric": "A quantifiable measure used to track and assess the performance of a specific aspect of an experiment. In Warehouse Native, metrics are defined in Harness FME.",
+ "experiment": "A controlled test to evaluate the impact of different variations on user behavior or system performance. In Warehouse Native, experiments are defined in Harness FME.",
+ "assignment-source": "A data source that defines how users are assigned to different variations in an experiment. In Warehouse Native, this data is typically stored in your data warehouse.",
+ "metric-source": "A data source that defines how metrics are collected and calculated for an experiment. In Warehouse Native, this data is typically stored in your data warehouse.",
+ "data-warehouse": "A centralized repository for storing and managing large volumes of structured and semi-structured data. Examples include Snowflake, BigQuery, Redshift, and Databricks.",
+ "warehouse-native": "A method of running feature management experiments directly within your data warehouse, leveraging its processing power and existing data infrastructure.",
+ "cloud-experimentation": "A method of running feature management experiments using a cloud-based service provided by Harness FME, which handles data collection, analysis, and reporting."
+ }
}
}
\ No newline at end of file
diff --git a/src/css/custom.css b/src/css/custom.css
index d8075e8deef..52c2fb2f091 100644
--- a/src/css/custom.css
+++ b/src/css/custom.css
@@ -1889,3 +1889,42 @@ html[data-theme='dark'] .sidebar-opensource > a::before {
background: linear-gradient(135deg, #3dc7f6, #00ade4);
box-shadow: 0 2px 4px rgba(61, 199, 246, 0.4);
}
+
+.sidebar-item-beta > .menu__list-item-collapsible > .menu__link {
+ position: relative;
+ display: flex !important;
+ align-items: center !important;
+ justify-content: space-between !important;
+}
+
+.sidebar-item-beta > .menu__list-item-collapsible > .menu__link::after {
+ content: 'BETA';
+ background: linear-gradient(135deg, #5a00cc, #ff00cc);
+ color: white;
+ font-size: 8px;
+ font-weight: 700;
+ padding: 2px 6px;
+ border-radius: 10px;
+ text-transform: uppercase;
+ letter-spacing: 0.5px;
+ margin-left: auto;
+ box-shadow: 0 2px 6px rgba(161, 0, 255, 0.4);
+ animation: betaBadgePulse 3s ease-in-out infinite;
+ flex-shrink: 0;
+}
+
+@keyframes betaBadgePulse {
+ 0%, 100% {
+ opacity: 1;
+ transform: scale(1);
+ }
+ 50% {
+ opacity: 0.85;
+ transform: scale(0.97);
+ }
+}
+
+[data-theme='dark'] .sidebar-item-beta > .menu__list-item-collapsible > .menu__link::after {
+ background: linear-gradient(135deg, #c458ff, #ff66e8);
+ box-shadow: 0 2px 6px rgba(255, 102, 232, 0.5);
+}
\ No newline at end of file
diff --git a/static/provider-logos/whn-integrations/bigquery-logo.svg b/static/provider-logos/whn-integrations/bigquery-logo.svg
new file mode 100644
index 00000000000..4ee5458647f
--- /dev/null
+++ b/static/provider-logos/whn-integrations/bigquery-logo.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/static/provider-logos/whn-integrations/redshift-logo.png b/static/provider-logos/whn-integrations/redshift-logo.png
new file mode 100644
index 00000000000..c3b8476b32f
Binary files /dev/null and b/static/provider-logos/whn-integrations/redshift-logo.png differ
diff --git a/static/provider-logos/whn-integrations/snowflake-logo.png b/static/provider-logos/whn-integrations/snowflake-logo.png
new file mode 100644
index 00000000000..81bf2d23880
Binary files /dev/null and b/static/provider-logos/whn-integrations/snowflake-logo.png differ
diff --git a/static/provider-logos/whn-integrations/trino-logo.png b/static/provider-logos/whn-integrations/trino-logo.png
new file mode 100644
index 00000000000..f1612909c1b
Binary files /dev/null and b/static/provider-logos/whn-integrations/trino-logo.png differ