Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions .changeset/grapheme-counter-entry.md

This file was deleted.

64 changes: 18 additions & 46 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,6 @@ countGraphemes('a̐éö̲');
> `countGraphemes()` is a small wrapper around `graphemeSegments()`.
>
> If you need it more than once at a time, consider memoization or use `graphemeSegments()` or `splitGraphemes()` once instead.
>
> If counting is on your hot path, use the standalone [`unicode-segmenter/grapheme-counter`](#export-unicode-segmentergrapheme-counter) entry instead.

#### Example: Build an advanced grapheme matcher

Expand Down Expand Up @@ -125,31 +123,6 @@ function* matchEmoji(str) {

Or build even more advanced one like an Unicode-aware [TTY string width](https://github.com/cometkim/unicode-string-width) utility.

### Export `unicode-segmenter/grapheme-counter`

A standalone counter for extended grapheme clusters.

The result is identical to `countGraphemes()` of `unicode-segmenter/grapheme`, but it runs the boundary rules directly without allocating segment objects or slicing strings, so counting is 4+ times faster and zero heap usage.

It is a separate entry so that counting doesn't have to carry the segmenter code, and vice versa.

#### Example: Count graphemes fast

```js
import { countGraphemes } from 'unicode-segmenter/grapheme-counter';

countGraphemes('👋 안녕!');
// => 5
```

> [!NOTE]
> The counter module contains duplicate code from `graphemeSegments()`.
>
> If you want a smaller bundle size and performance multipliers are not very important, use the `unicode-segmenter/grapheme` module instead.
>
> Full Segmenter is already optimized enough.


### Export `unicode-segmenter/intl-adapter`

[`Intl.Segmenter`] API adapter (only `granularity: "grapheme"` available yet)
Expand Down Expand Up @@ -242,32 +215,31 @@ Since [Hermes doesn't support the `Intl.Segmenter` API](https://github.com/faceb

#### JS Bundle Stats

| Name | Unicode® | ESM? | Size | Size (min) | Size (min+gzip) | Size (min+br) | Size (min+zstd) |
|-------------------------------------------------|----------|------|--------:|-----------:|----------------:|--------------:|----------------:|
| `unicode-segmenter/grapheme` | 17.0.0 | ✔️ | 9,009 | 5,171 | 2,497 | 2,249 | 2,544 |
| `unicode-segmenter/grapheme-counter` | 17.0.0 | ✔️ | 7,912 | 4,852 | 2,353 | 2,120 | 2,396 |
| `unicode-segmenter/grapheme + grapheme-counter` | 17.0.0 | ✔️ | 9,985 | 5,489 | 2,571 | 2,310 | 2,612 |
| `graphemer` | 15.0.0 | ✖️ ️ | 410,435 | 95,104 | 15,752 | 10,660 | 15,911 |
| `grapheme-splitter` | 10.0.0 | ✖️ | 122,254 | 23,682 | 7,852 | 4,802 | 6,753 |
| `@formatjs/intl-segmenter`* | 17.0.0 | ✖️ | 268,301 | 176,759 | 45,988 | 31,701 | 45,370 |
| `unicode-segmentation`* | 15.1.0 | - | 56,529 | 52,439 | 24,108 | 17,343 | 24,375 |
| `Intl.Segmenter`* | - | - | 0 | 0 | 0 | 0 | 0 |

| Name | Unicode® | ESM? | Size | Size (min) | Size (min+gzip) | Size (min+br) | Size (min+zstd) |
|--------------------------------------|----------|------|--------:|-----------:|----------------:|--------------:|----------------:|
| `unicode-segmenter/grapheme` | 17.0.0 | ✔️ | 8,782 | 5,069 | 2,462 | 2,203 | 2,507 |
| `unicode-segmenter/grapheme` (full*) | 17.0.0 | ✔️ | 9,268 | 5,313 | 2,624 | 2,350 | 2,672 |
| `graphemer` | 15.0.0 | ✖️ | 410,435 | 95,104 | 15,752 | 10,660 | 15,911 |
| `grapheme-splitter` | 10.0.0 | ✖️ | 122,254 | 23,682 | 7,852 | 4,802 | 6,753 |
| `@formatjs/intl-segmenter`* | 17.0.0 | ✖️ | 268,301 | 176,759 | 45,988 | 31,701 | 45,370 |
| `unicode-segmentation`* | 15.1.0 | - | 56,529 | 52,439 | 24,108 | 17,343 | 24,375 |
| `Intl.Segmenter`* | - | - | 0 | 0 | 0 | 0 | 0 |

* `unicode-segmenter/grapheme` provides count/split-only API which has much less overhead than full `Segmenter` implementation, which includes some redundant codes but tree-shakable.
* `@formatjs/intl-segmenter` handles grapheme, word, and sentence, but it's not tree-shakable.
* `unicode-segmentation` size contains only minimum WASM binary and its bindings to execute benchmarking. It will increases to expose more features.
* `Intl.Segmenter`'s Unicode data depends on the host, and may not be up-to-date.
* `Intl.Segmenter` may not be available in [some old browsers](https://caniuse.com/mdn-javascript_builtins_intl_segmenter), edge runtimes, or embedded environments.

#### Hermes Bytecode Stats

| Name | Bytecode size | Bytecode size (gzip)* |
|-------------------------------------------------|--------------:|----------------------:|
| `unicode-segmenter/grapheme` | 19,721 | 10,874 |
| `unicode-segmenter/grapheme-counter` | 18,210 | 10,069 |
| `unicode-segmenter/grapheme + grapheme-counter` | 20,667 | 11,371 |
| `graphemer` | 134,085 | 31,770 |
| `grapheme-splitter` | 63,942 | 19,165 |
| `@formatjs/intl-segmenter` | 329,547 | 136,751 |
| Name | Bytecode size | Bytecode size (gzip)* |
|-------------------------------------|--------------:|----------------------:|
| `unicode-segmenter/grapheme` | 19,196 | 10,715 |
| `unicode-segmenter/grapheme` (full) | 19,401 | 10,848 |
| `graphemer` | 134,085 | 31,770 |
| `grapheme-splitter` | 63,942 | 19,165 |
| `@formatjs/intl-segmenter` | 329,547 | 136,751 |

* The installation size contains _compressed_ assets.

Expand Down
26 changes: 13 additions & 13 deletions benchmark/_helper.js
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ export function printStats(reports) {
let value = report[key];
if (value == null) return '';
if (typeof value === 'number') return value.toLocaleString('en-US');
return i === 0 ? `\`${value}\`` : String(value);
return i === 0 ? `${value}` : String(value);
}));

let widths = columns.map((key, i) => {
Expand All @@ -92,10 +92,13 @@ export function printStats(reports) {
* `UPDATE_README=true`.
*
* The table is located by its `#### <section>` heading. Rows are
* matched by library name and columns by header text, both ignoring
* the `*` footnote marks, which are preserved as-is. Cells with no
* matching report column (e.g. Unicode®, ESM?) and rows with no
* matching report (e.g. `Intl.Segmenter`) are left untouched.
* matched by their first cell against `report.name` verbatim — names
* are expected to already contain any backticks/footnote marks (e.g.
* `` `unicode-segmenter/grapheme` (full*) ``), so the README row's
* name cell is preserved as-is. Columns are matched by header text
* with `*` footnote marks stripped. Cells with no matching report
* column (e.g. Unicode®, ESM?) and rows with no matching report
* (e.g. `Intl.Segmenter`) are left untouched.
*
* @param {Array<Record<string, unknown>>} reports
* @param {string} section heading text, e.g. 'JS Bundle Stats'
Expand All @@ -121,7 +124,10 @@ export async function updateReadmeStats(reports, section) {
let end = head + 1;
while (end < lines.length && lines[end].trimStart().startsWith('|')) end += 1;

let split = line => line.trim().replace(/^\||\|$/g, '').split('|').map(cell => cell.trim());
let split = line => line.trim().replace(/^\||\|$/g, '').split('|').map(cell => cell
.trim()
.replace(/\s+[\uFE0F\uFE0E]+$/, '')
.trim());
let stripMark = text => text.replace(/\*+$/, '').trim();

let header = split(lines[head]);
Expand All @@ -132,16 +138,10 @@ export async function updateReadmeStats(reports, section) {
let colKeys = header.map(stripMark);

let byName = new Map(reports.map(report => [String(report.name), report]));
for (let report of reports) {
// e.g. the 'unicode-segmentation (wasm-bindgen)' report matches
// the `unicode-segmentation` row
let alias = String(report.name).replace(/\s*\([^)]*\)$/, '');
if (!byName.has(alias)) byName.set(alias, report);
}

let matched = new Set();
for (let row of body) {
let report = byName.get(stripMark(row[0]).replace(/`/g, '').trim());
let report = byName.get(row[0]);
if (!report) continue;
matched.add(report);
for (let i = 1; i < header.length; i++) {
Expand Down
10 changes: 0 additions & 10 deletions benchmark/grapheme/bundle-entries/unicode-segmenter-both.js

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from '../../../src/grapheme.js';
2 changes: 0 additions & 2 deletions benchmark/grapheme/bundle-entries/unicode-segmenter.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
export {
graphemeSegments,
splitGraphemes,
countGraphemes,
} from '../../../src/grapheme.js';
11 changes: 5 additions & 6 deletions benchmark/grapheme/bundle-stats-hermes.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,11 @@ let config = await Metro.loadConfig(undefined, {
});

let libs = [
['unicode-segmenter/grapheme', 'bundle-entries/unicode-segmenter.js'],
['unicode-segmenter/grapheme-counter', 'bundle-entries/unicode-segmenter-counter.js'],
['unicode-segmenter/grapheme + grapheme-counter', 'bundle-entries/unicode-segmenter-both.js'],
['graphemer', 'bundle-entries/graphemer.js'],
['grapheme-splitter', 'bundle-entries/grapheme-splitter.js'],
['@formatjs/intl-segmenter', 'bundle-entries/formatjs-intl-segmenter.js'],
['`unicode-segmenter/grapheme`', 'bundle-entries/unicode-segmenter.js'],
['`unicode-segmenter/grapheme` (full)', 'bundle-entries/unicode-segmenter-full.js'],
['`graphemer`', 'bundle-entries/graphemer.js'],
['`grapheme-splitter`', 'bundle-entries/grapheme-splitter.js'],
['`@formatjs/intl-segmenter`', 'bundle-entries/formatjs-intl-segmenter.js'],
];

let reports = [];
Expand Down
13 changes: 6 additions & 7 deletions benchmark/grapheme/bundle-stats.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,11 @@ import { printStats, reportBundleStats, reportWasmBindingStats, updateReadmeStat
let baseDir = import.meta.dirname;

let libs = [
['unicode-segmenter/grapheme', 'bundle-entries/unicode-segmenter.js'],
['unicode-segmenter/grapheme-counter', 'bundle-entries/unicode-segmenter-counter.js'],
['unicode-segmenter/grapheme + grapheme-counter', 'bundle-entries/unicode-segmenter-both.js'],
['graphemer', 'bundle-entries/graphemer.js'],
['grapheme-splitter', 'bundle-entries/grapheme-splitter.js'],
['@formatjs/intl-segmenter', 'bundle-entries/formatjs-intl-segmenter.js'],
['`unicode-segmenter/grapheme`', 'bundle-entries/unicode-segmenter.js'],
['`unicode-segmenter/grapheme` (full*)', 'bundle-entries/unicode-segmenter-full.js'],
['`graphemer`', 'bundle-entries/graphemer.js'],
['`grapheme-splitter`', 'bundle-entries/grapheme-splitter.js'],
['`@formatjs/intl-segmenter`*', 'bundle-entries/formatjs-intl-segmenter.js'],
];

let reports = await Promise.all(
Expand All @@ -41,7 +40,7 @@ let reports = await Promise.all(
);

{
let libName = 'unicode-segmentation (wasm-bindgen)';
let libName = '`unicode-segmentation`*';
let bindingEntry = path.join(baseDir, 'bundle-entries/unicode-segmentation.js');
let bindingResult = await build({
write: false,
Expand Down
2 changes: 1 addition & 1 deletion benchmark/grapheme/memory-stats.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ function drain(iterable) {
/** @type {Record<string, import('../_memory.js').LibAdapter>} */
let libs = {
'unicode-segmenter/grapheme': async () => {
let { graphemeSegments } = await import('../../src/grapheme.js');
let { graphemeSegments } = await import('unicode-segmenter/grapheme');
return input => drain(graphemeSegments(input));
},
'graphemer': async () => {
Expand Down
5 changes: 0 additions & 5 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
"./emoji": "./src/emoji.js",
"./general": "./src/general.js",
"./grapheme": "./src/grapheme.js",
"./grapheme-counter": "./src/grapheme-counter.js",
"./utils": "./src/utils.js",
"./intl-adapter": {
"types": "./src/intl-adapter.js",
Expand Down Expand Up @@ -67,10 +66,6 @@
"import": "./grapheme.js",
"require": "./grapheme.cjs"
},
"./grapheme-counter": {
"import": "./grapheme-counter.js",
"require": "./grapheme-counter.cjs"
},
"./utils": {
"import": "./utils.js",
"require": "./utils.cjs"
Expand Down
Loading
Loading