diff --git a/documentation/reference/api/java-embedded.md b/documentation/reference/api/java-embedded.md index aea96c190..b6aba3ef3 100644 --- a/documentation/reference/api/java-embedded.md +++ b/documentation/reference/api/java-embedded.md @@ -31,6 +31,7 @@ import TabItem from "@theme/TabItem" questdb ${release.name} + `} ) @@ -55,59 +56,124 @@ import TabItem from "@theme/TabItem" -## Writing data -This section provides example codes to write data to WAL and non-WAL tables. See -[Write Ahead Log](/docs/concept/write-ahead-log) for details about the -differences between WAL and non-WAL tables. +## Overview -The following writers are available for data ingestion: +QuestDB is composed of three major packages: -- `WalWriter` for WAL tables -- `TableWriter` for non-WAL tables -- `TableWriterAPI` for both WAL and non-WAL tables as it is an interface for - `WalWriter` and `Table Writer` +- `cairo`, a custom storage engine which handles reading and writing data to disk, underpinning all database operations. +The underlying native format is [colummnar](/concept/storage-model), and optimised for time-series workloads. +- `griffin`, a custom query engine, which provides a rich SQL layer to interact with the data. +- `cutlass`, a custom network I/O stack, which provides TCP, HTTP, PG Wire, and other endpoints to write and read data. + +QuestDB is written for performance, and therefore diverges from more traditional Java projects. Instead, the database uses off-heap memory, +object pooling, vectorised execution, and other key techniques to ensure that each of these packages is as performant as possible. Some key features +are written in native code directly, using C/C++/Rust - for example, the JIT compiler for WHERE filters, and Parquet encoding. + +When working with QuestDB as an embedded storage engine, it is important to understand this and ensure that any shims or APIs you +add in front of the database a similarly performant, or else you will not be able to get the most out of it. + + +### High-level API + + +When using QuestDB as an embedded storage engine, the first class you will interact with is [`CairoEngine`](https://github.com/questdb/questdb/blob/master/core/src/main/java/io/questdb/cairo/CairoEngine.java) class. + +`CairoEngine` represents a specific database instance, with a root directory. Only a single engine can own a data directory at any one time. + +As the central context, `CairoEngine` becomes the root for your storage and querying operations. + +`cairo` is the custom storage engine that handles reading and writing data to disk. `cairo` is combined with `griffin` , our SQL query engine, +to add rich SQL support on top. `cutlass` is an additional package that handles external API and network I/O. + +In order + +### Creating tables + +There are several ways you can create a table in QuestDB. However, the most straightforward method is to use +the SQL interface to execute a DDL. + +Queries are executed using a `SqlExecutionContext`. This is a context for compiling and running a particular query. + +Therefore, to create a table, we will instruct `CairoEngine` a `CREATE TABLE` statement within a particular `SqlExecutionContext`. + +```java title="Creating a table" +// Create a configuration for the engine. +final CairoConfiguration configuration = new DefaultCairoConfiguration("/path/to/dbroot"); + +// Initialise the engine +try (CairoEngine engine = new CairoEngine(configuration)) { + // Create a new query execution context + try (SqlExecutionContext executionContext = + new SqlExecutionContextImpl(engine, 1) // single-threaded query execution + .with(AllowAllSecurityContext.INSTANCE, null)) // query can access any table) + { + // Execute the create table statement + engine.execute( + "CREATE TABLE 'trades' (" + + "symbol SYMBOL CAPACITY 256 CACHE, " + + "side SYMBOL CAPACITY 256 CACHE, " + + "price DOUBLE, " + + "amount DOUBLE, " + + "timestamp TIMESTAMP" + + ") timestamp(timestamp) PARTITION BY DAY WAL", + executionContext + ); + } catch (SqlException e) { + throw new RuntimeException(e); + } +} +``` + + +### Writing data + +QuestDB supports two main types of tables - WAL ([write-ahead-log](/docs/concept/write-ahead-log)) and non-WAL. Most users should use WAL tables for their time-series data, as WAL brings many benefits, notably concurrent writes from multiple APIs. + +Table writing is governed by the [`TableWriterAPI`](https://github.com/questdb/questdb/blob/master/core/src/main/java/io/questdb/cairo/TableWriterAPI.java) interface. + +This defines common functionality across both of the `WalWriter` (WAL tables) and `TableWriter` (non-WAL tables) classes. + +This API cannot be used to create a table - the table must already exist. + +In the prior example, we created a WAL table, so we will now write to it using `WalWriter`. ### Writing data using `WalWriter` -The `WalWriter` facilitates table writes to WAL tables. To successfully create -an instance of `WalWriter`, the table must already exist. +`WalWriter` first writes the data into a write-head-log file. Then, the `ApplyWal2TableJob` will later commit this data +to the table, making it visible for read. + +In a standalone QuestDB deployment, a thread-pool automatically executes `ApplyWal2TableJob`, keeping your tables up to date. + +For this example, we will create and execution a job to move the data from the WAL to the table. + + ```java title="Example WalWriter" -final CairoConfiguration configuration = new DefaultCairoConfiguration("data_dir"); -try (CairoEngine engine = new CairoEngine(configuration)) { - final SqlExecutionContext ctx = new SqlExecutionContextImpl(engine, 1) - .with(AllowAllSecurityContext.INSTANCE, null); - engine.ddl("CREATE TABLE testTable (" + - "a int, b byte, c short, d long, e float, g double, h date, " + - "i symbol, j string, k boolean, l geohash(8c), ts timestamp" + - ") TIMESTAMP(ts) PARTITION BY DAY WAL", ctx); +// - // write data into WAL - final TableToken tableToken = engine.getTableTokenIfExists("testTable"); - try (WalWriter writer = engine.getWalWriter(tableToken)) { - for (int i = 0; i < 3; i++) { - TableWriter.Row row = writer.newRow(Os.currentTimeMicros()); - row.putInt(0, 123); - row.putByte(1, (byte) 1111); - row.putShort(2, (short) 222); - row.putLong(3, 333); - row.putFloat(4, 4.44f); - row.putDouble(5, 5.55); - row.putDate(6, System.currentTimeMillis()); - row.putSym(7, "xyz"); - row.putStr(8, "abc"); - row.putBool(9, true); - row.putGeoHash(10, GeoHashes.fromString("u33dr01d", 0, 8)); - row.append(); - } - writer.commit(); - } +// first, acquire a handle for the table +final TableToken tableToken = engine.getTableTokenIfExists("trades"); - // apply WAL to the table - try (ApplyWal2TableJob walApplyJob = new ApplyWal2TableJob(engine, 1, 1)) { - while (walApplyJob.run(0)) ; +// get a WalWriter from the CairoEngine's object pool +try (WalWriter writer = engine.getWalWriter(tableToken)) { + // append rows to the table + for (int i = 0; i < 3; i++) { + TableWriter.Row row = writer.newRow(Os.currentTimeMicros()); + row.putSym(0, "ETH-USD"); + row.putSym(1, "sell"); + row.putDouble(2, 2615.54); + row.putDouble(3, 0.00044); + row.append(); } + // commit the data to the write-ahead-log + writer.commit(); +} + +// apply WAL to the table +try (ApplyWal2TableJob walApplyJob = + new ApplyWal2TableJob(engine, 1)) { + while (walApplyJob.run(0)) ; } ```