Skip to content

docs: add Getting Started section #4143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dahlinomine wants to merge 1 commit into
base: master
Choose a base branch
from
+10 −129
Open

docs: add Getting Started section #4143

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 10 additions & 129 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,15 @@ and helps data platforms or products easily build infra-decoupled, stream-and-ba

Learn more about Amoro at https://amoro.apache.org/, contact the developers and community on the [mailing list](https://amoro.apache.org/join-community/#mailing-lists) if you need any help.

## Getting Started

The quickest way to experience Amoro is to use the [Quick Start Guide](https://amoro.apache.org/docs/latest/quickstart/setup/). You can also start the Amoro Management Service (AMS) and its components using Docker:

```bash
docker run -p 1630:1630 -p 1260:1260 neteaseamoro/amoro:latest
```
Access the dashboard at `http://localhost:1630` (Default: admin/admin).

## Architecture

Here is the architecture diagram of Amoro:
Expand All @@ -54,132 +63,4 @@ Here is the architecture diagram of Amoro:
* AMS: Amoro Management Service provides Lakehouse management features, like self-optimizing, data expiration, etc.
It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services.
* Plugins: Amoro provides a wide selection of external plugins to meet different scenarios.
* Optimizers: The self-optimizing execution engine plugin asynchronously performs merging, sorting, deduplication,
Copy link
Copy Markdown
Member

@klion26 klion26 Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we remove this?

layout optimization, and other operations on all type table format tables.
* Terminal: SQL command-line tools, provide various implementations like local Spark and Kyuubi.
* LogStore: Provide millisecond to second level SLAs for real-time data processing based on message queues like Kafka and Pulsar.

## Supported table formats

Amoro can manage tables of different table formats, similar to how MySQL/ClickHouse can choose different storage engines.
Amoro meets diverse user needs by using different table formats. Currently, Amoro supports four table formats:

* Iceberg format: Users can directly entrust their Iceberg tables to Amoro for maintenance, so that users can not only use all the functions of Iceberg tables, but also enjoy the performance and stability improvements brought by Amoro.
* Mixed-Iceberg format: Amoro provides a set of more optimized formats for streaming update scenarios on top of the Iceberg format. If users have high performance requirements for streaming updates or have demands for CDC incremental data reading functions, they can choose to use the Mixed-Iceberg format.
* Mixed-Hive format: Many users do not want to affect the business originally built on Hive while using data lakes. Therefore, Amoro provides the Mixed-Hive format, which can upgrade Hive tables to Mixed-Hive format only through metadata migration, and the original Hive tables can still be used normally. This ensures business stability and benefits from the advantages of data lake computing.
* Paimon format: Amoro supports displaying metadata information in the Paimon format, including Schema, Options, Files, Snapshots, DDLs, and Compaction information.

## Supported engines

### Iceberg format

Iceberg format tables use the engine integration method provided by the Iceberg community.
For details, please refer to: [Iceberg Docs](https://iceberg.apache.org/docs/latest/).

### Mixed format

Amoro support multiple processing engines for Mixed format as below:

| Processing Engine | Version | Batch Read | Batch Write | Batch Overwrite | Streaming Read | Streaming Write | Create Table | Alter Table |
|-------------------|------------------------|-------------|-------------|-----------------|----------------|-----------------|--------------|-------------|
| Flink | 1.17.x, 1.18.x | ✔ | ✔ | ✖ | ✔ | ✔ | ✔ | ✖ |
| Spark | 3.3, 3.4, 3.5 | ✔ | ✔ | ✔ | ✖ | ✖ | ✔ | ✔ |
| Hive | 2.x, 3.x | ✔ | ✖ | ✔ | ✖ | ✖ | ✖ | ✔ |
| Trino | 406 | ✔ | ✖ | ✔ | ✖ | ✖ | ✖ | ✔ |

## Features

- Self-optimizing - Continuously optimizing tables, including compacting small files, change files, regularly delete expired files to keep high query performance and reducing storage costs.
- Multiple Formats - Support different table formats such as Iceberg, Mixed-Iceberg and Mixed-Hive to meet different scenario requirements and provide them with unified management capabilities.
- Catalog Service - Provide a unified catalog service for all compute engines, which can also used with existing metadata store service such as Hive Metastore and AWS Glue.
- Rich Plugins - Provide various plugins to integrate with other systems, like continuously optimizing with Flink and data analysis with Spark and Kyuubi.
- Management Tools - Provide a variety of management tools, including WEB UI and standard SQL command line, to help you get started faster and integrate with other systems more easily.
- Infrastructure Independent - Can be easily deployed and used in private environments, cloud environments, hybrid cloud environments, and multi-cloud environments.

## Modules

Amoro contains modules as below:

- `amoro-common` contains core abstractions and common implementation for other modules
- `amoro-ams` is amoro management service module
- `amoro-web` is the dashboard frontend for ams
- `amoro-optimizer` provides default optimizer implementation
- `amoro-format-iceberg` contains integration of Apache Iceberg format
- `amoro-format-hudi` contains integration of Apache Hudi format
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive format
- `amoro-mixed-flink` provides Flink connectors for Mixed format tables (use amoro-flink-runtime for a shaded version)
- `amoro-mixed-spark` provides Spark connectors for Mixed format tables (use amoro-spark-runtime for a shaded version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables


## Building

Amoro is built using Maven with JDK 8, 11 and 17(required for `amoro-format-mixed/amoro-mixed-trino` module).

* Build all modules without `amoro-mixed-trino`: `./mvnw clean package`
* Build and skip tests: `./mvnw clean package -DskipTests`
* Build and skip dashboard: `./mvnw clean package -Pskip-dashboard-build`
* Build and disable disk storage, RocksDB will NOT be introduced to avoid memory overflow: `./mvnw clean package -DskipTests -Pno-extented-disk-storage`
* Build and enable aliyun-oss-sdk: `./mvnw clean package -DskipTests -Paliyun-oss-sdk`
* Build with hadoop 2.x(the default is 3.x) dependencies: `./mvnw clean package -DskipTests -Phadoop2`
* Specify Flink version for Flink optimizer(the default is 1.20.0): `./mvnw clean package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
* If the version of Flink is below 1.15.0, you also need to add the `-Pflink-optimizer-pre-1.15` parameter: `./mvnw clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
* Specify Spark version for Spark optimizer(the default is 3.5.7): `./mvnw clean package -DskipTests -Dspark.version=3.5.7`
* Build `amoro-mixed-trino` module under JDK 17: `./mvnw clean package -DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl 'amoro-format-mixed/amoro-mixed-trino' -am`.
* Build all modules: `./mvnw clean package -DskipTests -Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml` in `${user.home}/.m2/` dir with content below.
* Build a distribution package with all formats integrated: `./mvnw clean package -Psupport-all-formats`
* Build a distribution package with Apache Paimon format: `./mvnw clean package -Psupport-paimon-format`
* Build a distribution package with Apache Hudi format: `./mvnw clean package -Psupport-hudi-format`

```
<?xml version="1.0" encoding="UTF-8"?>
<toolchains>
<toolchain>
<type>jdk</type>
<provides>
<version>17</version>
<vendor>sun</vendor>
</provides>
<configuration>
<jdkHome>${YourJDK17Home}</jdkHome>
</configuration>
</toolchain>
</toolchains>
```

## Quickstart

Visit [https://amoro.apache.org/quick-start/](https://amoro.apache.org/quick-start/) to quickly
explore what amoro can do.

## Join Community

If you are interested in Lakehouse, Data Lake Format, welcome to join our community, we welcome any organizations, teams
and individuals to grow together, and sincerely hope to help users better use Data Lake Format through open source.

### Slack

You can join the Amoro community on Slack. Amoro channel is in ASF Slack workspace.

- Anyone with an @apache.org email address can become a full member of the ASF Slack workspace.
- Search [Amoro channel](https://the-asf.slack.com/archives/C06RZ9UHUTH) and join it.
- If you don't have an @apache.org email address, you can email to `dev@amoro.apache.org` to apply for an
[ASF Slack invitation](https://infra.apache.org/slack.html). Then join [Amoro channel](https://the-asf.slack.com/archives/C06RZ9UHUTH).

### Wechat
Join the Amoro WeChat Group: Add " `kllnn999` " as a friend and request to join the group.

## Contributors
This project exists thanks to all the people who contribute.

<a href="https://github.com/apache/amoro/graphs/contributors">
<img src="https://contrib.rocks/image?repo=apache/amoro" />
</a>

Made with [contrib.rocks](https://contrib.rocks).

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=apache/amoro&type=Date)](https://star-history.com/#apache/amoro&Date)
* Optimizers: The self-optimizing execution engine plugin asynchronously performs merging, sorting, deduplication,
Loading