diff --git a/docs/user-guide/logs/json2.md b/docs/user-guide/logs/json2.md new file mode 100644 index 0000000000..f629d818d9 --- /dev/null +++ b/docs/user-guide/logs/json2.md @@ -0,0 +1,262 @@ +--- +keywords: [JSON2, JSON, logs, structured logs, type hints, json_get] +description: Learn how to use the JSON2 type in GreptimeDB, including table creation, type hints, JSON field access, and current limitations. +--- + +# JSON2 Type + +JSON2 is a JSON type in GreptimeDB designed for logs and semi-structured data. +It stores fields inside JSON in a structured, columnar form so that frequently +used fields can be read, filtered, and aggregated efficiently like regular +columns, while still preserving the flexibility of JSON for dynamic schemas. + +:::note +JSON2 is currently in Beta, and some capabilities are still being improved. +::: + +## Quick Start + +The following example creates an API access log table, inserts a few request +logs, and queries fields from JSON2. Fixed fields are stored in regular columns, +while fields in `attrs` use JSON2 because their structure may vary but they are +still queried frequently. + +### Create a table + +When creating a table, you can declare a JSON2 column with the `JSON2` type. +Currently, JSON2 can only be used in append-only tables, so you must set +`'append_mode' = 'true'` when creating the table. + +```sql +CREATE TABLE application_logs ( + ts TIMESTAMP TIME INDEX, + app_name STRING, + log_level STRING, + `message` STRING, + attrs JSON2, +) WITH ( + 'append_mode' = 'true' +); +``` + +### Insert JSON data + +When writing to a JSON2 column, you can insert a JSON object. The following data +includes one successful request, one slow request, and one failed request: + +```sql +INSERT INTO application_logs +VALUES + ( + 1, + 'checkout', + 'INFO', + 'request completed', + '{"trace_id":"8f3a1c","user":{"id":1001,"name":"Alice"},"http":{"method":"POST","path":"/v1/orders","status":200},"latency_ms":42.8}' + ), + ( + 2, + 'checkout', + 'WARN', + 'slow request', + '{"trace_id":"8f3a1d","user":{"id":1002,"name":"Bob"},"http":{"method":"POST","path":"/v1/orders","status":200},"latency_ms":386.4}' + ), + ( + 3, + 'checkout', + 'ERROR', + 'request failed', + '{"trace_id":"8f3a1e","user":{"id":1003},"http":{"method":"POST","path":"/v1/orders","status":500},"latency_ms":71.2,"error":true}' + ); +``` + +### Query JSON fields + +You can read fields from JSON2 directly with dot paths: + +```sql +SELECT + ts, + app_name, + attrs.trace_id AS trace_id, + attrs.user.name AS user_name, + attrs.http.status AS status, + attrs.latency_ms AS latency_ms, + attrs.error AS error +FROM application_logs +ORDER BY ts; +``` + +The query result is: + +| ts | app_name | trace_id | user_name | status | latency_ms | error | +| --- | --- | --- | --- | --- | --- | --- | +| 1970-01-01 00:00:00.001 | checkout | 8f3a1c | Alice | 200 | 42.8 | NULL | +| 1970-01-01 00:00:00.002 | checkout | 8f3a1d | Bob | 200 | 386.4 | NULL | +| 1970-01-01 00:00:00.003 | checkout | 8f3a1e | NULL | 500 | 71.2 | true | + +You can also use JSON functions and cast the return type explicitly: + +```sql +SELECT + json_get(attrs, 'http.path')::STRING AS path, + json_get(attrs, 'http.status')::INT8 AS status, + json_get(attrs, 'latency_ms')::DOUBLE AS latency_ms, + json_get(attrs, 'error')::BOOLEAN AS error +FROM application_logs +WHERE json_get(attrs, 'http.status')::INT8 >= 500 + OR json_get(attrs, 'latency_ms')::DOUBLE > 300 +ORDER BY ts; +``` + +The query result is: + +| path | status | latency_ms | error | +| --- | --- | --- | --- | +| /v1/orders | 200 | 386.4 | NULL | +| /v1/orders | 500 | 71.2 | true | + +You can also aggregate fields, for example to count requests, errors, and average +latency for each API path: + +```sql +SELECT + json_get(attrs, 'http.path')::STRING AS path, + COUNT(*) AS requests, + SUM(CASE WHEN json_get(attrs, 'error')::BOOLEAN THEN 1 ELSE 0 END) AS errors, + ROUND(AVG(json_get(attrs, 'latency_ms')::DOUBLE), 1) AS avg_latency_ms +FROM application_logs +GROUP BY json_get(attrs, 'http.path')::STRING; +``` + +The query result is: + +| path | requests | errors | avg_latency_ms | +| --- | --- | --- | --- | +| /v1/orders | 3 | 1 | 166.8 | + +## Syntax + +### JSON Field Type hints + +JSON2 supports type hints for declaring concrete data types for selected +subpaths. Type hints are recommended for frequently queried subpaths with known +and stable types. These subpaths are stored using the specified types, providing +query performance close to regular columns. JSON2 also validates their values +during writes. Type hints are optional. For subpaths without type hints, JSON2 +infers their types from the values written to the column. + +The syntax for declaring type hints is: + +```sql +json_column JSON2 ( + path.to.field DATA_TYPE [NULL | NOT NULL] [DEFAULT literal] +) +``` + +Type hint paths use dot notation. For example, `user.id` refers to the following +JSON path: `{"user":{"id":...}}`. + +If a JSON key itself contains a dot, wrap that path segment in double quotes. +For example, `"service.name"` means a key named `service.name` in the root object, +not a nested path `service.name`. + +Type hints currently support the following data types: + +- `STRING` +- `BIGINT` +- `BIGINT UNSIGNED` +- `DOUBLE` +- `BOOLEAN` + +Type hints allow `NULL` by default. If you specify `NOT NULL`, that path must +exist in the written JSON. + +You can declare type hints directly in the `CREATE TABLE` statement. The +following example defines type hints for commonly queried subpaths in the +`attrs` column: + +```sql +CREATE TABLE application_logs ( + ts TIMESTAMP TIME INDEX, + app_name STRING, + log_level STRING, + `message` STRING, + attrs JSON2 ( + trace_id STRING, + user.id BIGINT, + user.name STRING DEFAULT 'anonymous', + http.method STRING, + http.path STRING, + http.status BIGINT, + latency_ms DOUBLE, + error BOOLEAN DEFAULT false + ) +) WITH ( + 'append_mode' = 'true' +); +``` + +### `json_get` UDF + +`json_get` reads a nested field from JSON2 by path. It returns a string by +default. If you want to specify the return type directly, add a cast after the +function. + +The syntax of `json_get` is: + +```sql +json_get(json_column, 'path.to.field')::TYPE +``` + +`json_get` can be used in `SELECT`, `WHERE`, `GROUP BY`, and other SQL clauses +that accept expressions. For example: + +```sql +SELECT + json_get(attrs, 'trace_id')::STRING AS trace_id, + json_get(attrs, 'http.status')::BIGINT AS status, + json_get(attrs, 'latency_ms')::DOUBLE AS latency_ms +FROM application_logs +WHERE json_get(attrs, 'http.status')::BIGINT >= 500; +``` + +### Dot syntax + +You can read JSON2 subpaths directly with dot syntax: + +```sql +json_column.path.to.field +``` + +Dot syntax can be used in `SELECT`, `WHERE`, `GROUP BY`, and other SQL clauses +that accept expressions. For example: + +```sql +SELECT + attrs.trace_id, + attrs.http.status, + attrs.latency_ms +FROM application_logs +WHERE attrs.http.status >= 500; +``` + +## Roadmap + +JSON2 is currently in Beta and still has the following limitations. Future +releases will continue to improve these capabilities: + +- Support JSON2 in non-append-only tables. +- Support writing non-object or empty-object JSON root values such as arrays, + strings, numbers, booleans, `null`, and `{}`. +- Support querying the JSON2 root column itself. For now, query specific + subpaths such as `attrs.http.status` or `json_get(attrs, 'http.status')`. +- Support subscript access to elements inside JSON arrays. For now, you can + query `attrs.items`, but not `attrs.items[0]` or `json_get(attrs, 'items[0]')`. +- Support functions such as `json_get_string`, `json_get_int`, + `json_get_float`, and `json_get_bool` for JSON2. +- Extend supported type hint data types, such as time-related types like + `TIMESTAMP`. +- Support index options such as `INVERTED INDEX` and `SKIPPING INDEX` for type + hints. +- Support writing JSON2 through OTLP and other ingestion paths. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/json2.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/json2.md new file mode 100644 index 0000000000..3f7ed29cb3 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/json2.md @@ -0,0 +1,235 @@ +--- +keywords: [JSON2, JSON, logs, structured logs, type hints, json_get] +description: 介绍 GreptimeDB 中 JSON2 类型的使用方式,包括建表、type hint、JSON 字段读取以及当前限制。 +--- + +# JSON2 类型 + +JSON2 是 GreptimeDB 为日志和半结构化数据设计的 JSON 类型。 +它会将 JSON 中的字段以结构化、列式的方式存储,使常用字段能够像普通列一样被高效读取、过滤和聚合,同时保留 JSON 对动态结构的表达能力。 + +:::note +JSON2 目前处于 Beta 阶段,部分功能仍在持续完善中。 +::: + +## 快速入门 + +下面的示例创建一张 API 访问日志表,写入几条请求日志,并查询 JSON2 中的字段。固定字段放在普通列中,结构可能变化但经常查询的字段放在 JSON2 列 `attrs` 中。 + +### 创建表 + +在建表时,可以使用 `JSON2` 类型声明 JSON2 列。 +当前 JSON2 只能在 append-only 表中使用,因此建表时需要设置 `'append_mode' = 'true'`。 + +```sql +CREATE TABLE application_logs ( + ts TIMESTAMP TIME INDEX, + app_name STRING, + log_level STRING, + `message` STRING, + attrs JSON2, +) WITH ( + 'append_mode' = 'true' +); +``` + +### 写入 JSON 数据 + +写入 JSON2 列时,可以写入 JSON object。下面的数据包含一次成功请求、一次慢请求和一次失败请求: + +```sql +INSERT INTO application_logs +VALUES + ( + 1, + 'checkout', + 'INFO', + 'request completed', + '{"trace_id":"8f3a1c","user":{"id":1001,"name":"Alice"},"http":{"method":"POST","path":"/v1/orders","status":200},"latency_ms":42.8}' + ), + ( + 2, + 'checkout', + 'WARN', + 'slow request', + '{"trace_id":"8f3a1d","user":{"id":1002,"name":"Bob"},"http":{"method":"POST","path":"/v1/orders","status":200},"latency_ms":386.4}' + ), + ( + 3, + 'checkout', + 'ERROR', + 'request failed', + '{"trace_id":"8f3a1e","user":{"id":1003},"http":{"method":"POST","path":"/v1/orders","status":500},"latency_ms":71.2,"error":true}' + ); +``` + +### 查询 JSON 字段 + +可以直接通过点号路径读取 JSON2 中的字段: + +```sql +SELECT + ts, + app_name, + attrs.trace_id AS trace_id, + attrs.user.name AS user_name, + attrs.http.status AS status, + attrs.latency_ms AS latency_ms, + attrs.error AS error +FROM application_logs +ORDER BY ts; +``` + +查询结果如下: + +| ts | app_name | trace_id | user_name | status | latency_ms | error | +| --- | --- | --- | --- | --- | --- | --- | +| 1970-01-01 00:00:00.001 | checkout | 8f3a1c | Alice | 200 | 42.8 | NULL | +| 1970-01-01 00:00:00.002 | checkout | 8f3a1d | Bob | 200 | 386.4 | NULL | +| 1970-01-01 00:00:00.003 | checkout | 8f3a1e | NULL | 500 | 71.2 | true | + +也可以使用 JSON 函数直接指定返回类型: + +```sql +SELECT + json_get(attrs, 'http.path')::STRING AS path, + json_get(attrs, 'http.status')::INT8 AS status, + json_get(attrs, 'latency_ms')::DOUBLE AS latency_ms, + json_get(attrs, 'error')::BOOLEAN AS error +FROM application_logs +WHERE json_get(attrs, 'http.status')::INT8 >= 500 + OR json_get(attrs, 'latency_ms')::DOUBLE > 300 +ORDER BY ts; +``` + +查询结果如下: + +| path | status | latency_ms | error | +| --- | --- | --- | --- | +| /v1/orders | 200 | 386.4 | NULL | +| /v1/orders | 500 | 71.2 | true | + +你也可以对字段做聚合,例如统计每个 API 路径的请求量、错误数和平均延迟: + +```sql +SELECT + json_get(attrs, 'http.path')::STRING AS path, + COUNT(*) AS requests, + SUM(CASE WHEN json_get(attrs, 'error')::BOOLEAN THEN 1 ELSE 0 END) AS errors, + ROUND(AVG(json_get(attrs, 'latency_ms')::DOUBLE), 1) AS avg_latency_ms +FROM application_logs +GROUP BY json_get(attrs, 'http.path')::STRING; +``` + +查询结果如下: + +| path | requests | errors | avg_latency_ms | +| --- | --- | --- | --- | +| /v1/orders | 3 | 1 | 166.8 | + +## 语法 + +### JSON 字段 Type hint + +JSON2 支持使用 type hint 为指定的子路径声明确定的数据类型。对于类型已知且稳定、需要频繁查询的子路径,建议使用 type hint。这些子路径会按指定类型存储,从而获得接近普通列的查询性能。JSON2 还会在写入时校验这些子路径的值。Type hint 是可选的。对于未声明 type hint 的子路径,JSON2 会根据写入列中的值推断其类型。 + +声明 type hint 的语法如下: + +```sql +json_column JSON2 ( + path.to.field DATA_TYPE [NULL | NOT NULL] [DEFAULT literal] +) +``` + +Type hint 的路径使用点号分隔,例如 `user.id` 对应 JSON 中的 +`{"user":{"id":...}}`。 + +如果某个 JSON key 本身包含点号,需要用双引号包住该路径段。 +例如 `"service.name"` 表示读取 root object 中名为 `service.name` +的 key,而不是读取 `service.name` 这条嵌套路径。 + +当前 type hint 支持以下类型: + +- `STRING` +- `BIGINT` +- `BIGINT UNSIGNED` +- `DOUBLE` +- `BOOLEAN` + +Type hint 默认允许 `NULL`。如果设置 `NOT NULL`,写入的 JSON 中必须存在该路径。 + +可以直接在 `CREATE TABLE` 语句中声明 type hint。下面的示例为 `attrs` 列中经常查询的子路径定义了 type hint: + +```sql +CREATE TABLE application_logs ( + ts TIMESTAMP TIME INDEX, + app_name STRING, + log_level STRING, + `message` STRING, + attrs JSON2 ( + trace_id STRING, + user.id BIGINT, + user.name STRING DEFAULT 'anonymous', + http.method STRING, + http.path STRING, + http.status BIGINT, + latency_ms DOUBLE, + error BOOLEAN DEFAULT false + ) +) WITH ( + 'append_mode' = 'true' +); +``` + +### `json_get` UDF + +`json_get` 用于按路径读取 JSON2 中的嵌套字段。默认返回字符串类型;如果希望直接指定返回类型,可以在函数后使用类型转换。 + +`json_get` 的语法如下: + +```sql +json_get(json_column, 'path.to.field')::TYPE +``` + +`json_get` 可以用于 `SELECT`、`WHERE`、`GROUP BY` 等接受表达式的 SQL 子句。例如: + +```sql +SELECT + json_get(attrs, 'trace_id')::STRING AS trace_id, + json_get(attrs, 'http.status')::BIGINT AS status, + json_get(attrs, 'latency_ms')::DOUBLE AS latency_ms +FROM application_logs +WHERE json_get(attrs, 'http.status')::BIGINT >= 500; +``` + +### 点号语法 + +可以直接通过点号语法读取 JSON2 中的子路径: + +```sql +json_column.path.to.field +``` + +点号语法可以用于 `SELECT`、`WHERE`、`GROUP BY` 等接受表达式的 SQL 子句。例如: + +```sql +SELECT + attrs.trace_id, + attrs.http.status, + attrs.latency_ms +FROM application_logs +WHERE attrs.http.status >= 500; +``` + +## 未来规划 + +JSON2 目前处于 Beta 阶段,仍有以下限制。后续版本会继续完善相关能力: + +- 支持在非 append-only 表中使用 JSON2。 +- 支持写入 array、string、number、boolean、`null` 和 `{}` 等非 object 或空 object 的 JSON root 值。 +- 支持查询 JSON2 root 列本身。目前请查询具体子路径,例如 `attrs.http.status` 或 `json_get(attrs, 'http.status')`。 +- 支持通过下标访问 JSON array 中的元素。目前可以查询 `attrs.items`,但暂不支持 `attrs.items[0]` 或 `json_get(attrs, 'items[0]')`。 +- 支持 `json_get_string`、`json_get_int`、`json_get_float` 和 `json_get_bool` 等函数处理 JSON2 类型。 +- 扩展 type hint 支持的类型,例如 `TIMESTAMP` 等时间类型。 +- 为 type hint 支持 `INVERTED INDEX`、`SKIPPING INDEX` 等索引选项。 +- 支持通过 OTLP 和其他 ingestion path 写入 JSON2。 diff --git a/sidebars.ts b/sidebars.ts index beca2a6273..63db3a087d 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -238,6 +238,7 @@ const sidebars: SidebarsConfig = { label: 'Overview', }, 'user-guide/logs/quick-start', + 'user-guide/logs/json2', 'user-guide/logs/use-custom-pipelines', 'user-guide/logs/fulltext-search', 'user-guide/logs/manage-pipelines',