feat(markdown): unify tag syntax and recognition (#6132)
This commit is contained in:
parent
571e0a3ff6
commit
c68e3d5bf9
65 changed files with 12451 additions and 665 deletions
585
docs/adr/0001-tag-syntax-and-recognition.md
Normal file
585
docs/adr/0001-tag-syntax-and-recognition.md
Normal file
|
|
@ -0,0 +1,585 @@
|
|||
# ADR 0001: Tag Syntax and Recognition
|
||||
|
||||
Status: Accepted
|
||||
|
||||
Date: 2026-08-01
|
||||
|
||||
Domain glossary: [Memos domain glossary](../glossary.md)
|
||||
|
||||
## Context
|
||||
|
||||
Tags are part of Memos' Markdown language, not merely an editor decoration. A tag occurrence affects rendered memo content, the derived tags stored in a
|
||||
memo payload, API responses, tag counts, filters, tag metadata, navigation, and editor completion.
|
||||
|
||||
Memos currently has four related but different implementations:
|
||||
|
||||
- The Go Markdown parser recognizes Unicode letters, marks, numbers, and symbols, plus `ZWJ`, `_`, `-`, `/`, and `&`. It extracts at most 100 Unicode
|
||||
code points and has no general left boundary.
|
||||
- The read-only remark plugin uses a similar frontend character class without `ZWJ`. It rejects a run longer than 100 code points and has special handling
|
||||
for adjacent `#` characters.
|
||||
- The editor decoration uses the frontend character class but requires a non-letter/non-number before `#`, so it does not highlight `hello#tag`. It scans
|
||||
raw text without consulting the editor's Markdown syntax tree.
|
||||
- Editor completion has no left boundary, no length limit, and no Markdown-context check.
|
||||
|
||||
These differences expose implementation drift but do not define the desired language. This ADR starts from intended product semantics; current behavior
|
||||
is non-normative.
|
||||
|
||||
This ADR defines the current Memos tag syntax and recognition rules, covering:
|
||||
|
||||
- The lexical form of a tag identifier.
|
||||
- How a tag candidate is found in flowing Markdown text.
|
||||
- Which Markdown contexts can contain a tag occurrence.
|
||||
- Pinned Unicode and emoji data.
|
||||
|
||||
Existing-data migration, rollout, backfill, and metadata remapping are outside this ADR's scope.
|
||||
|
||||
Markdown context is defined against the [GFM 0.29-gfm specification](https://github.github.com/gfm/). Memos-specific Markdown extensions layer on that
|
||||
baseline and are opaque to tag recognition unless their own definition explicitly exposes ordinary text.
|
||||
|
||||
The slash is a hierarchy separator between non-empty segments. `-`, `+`, and `&` are unrestricted visible segment units. Tag equality is exact: the display
|
||||
value itself is the comparison key, with no case folding or Unicode normalization.
|
||||
|
||||
A tag is a value derived from memo Markdown, not a durable entity with an identity independent of its source occurrences. Memos cannot rename a tag by
|
||||
changing metadata or a separate tag record. Renaming `#Work` to `#work` across 100 memos requires editing those 100 memo sources.
|
||||
|
||||
## Decision drivers
|
||||
|
||||
- Work naturally for multilingual personal notes, including numeric tags and emoji.
|
||||
- Recognize `hello#tag` intentionally without mistaking URL fragments for tags.
|
||||
- Give the backend, renderer, editor decoration, and completion the same values and source spans.
|
||||
- Use Markdown syntax rather than ad hoc URL or code regular expressions.
|
||||
- Avoid accepting every Unicode symbol merely to support emoji.
|
||||
- Keep parsing deterministic across Go, JavaScript runtimes, and Unicode upgrades.
|
||||
- Keep lexical scanning linear in memo size.
|
||||
|
||||
## Terminology
|
||||
|
||||
**Tag**
|
||||
: A classification value in a memo tag set, derived from one or more tag occurrences either as a direct tag value or as an implied ancestor. A tag is not
|
||||
independently created, owned, or renamed outside the source text that produces it.
|
||||
|
||||
**Introducer**
|
||||
: An ASCII number sign `#` (U+0023) that is not part of a matched fully-qualified emoji sequence and begins a tag candidate. It is not included in the
|
||||
extracted tag value.
|
||||
|
||||
**Tag identifier**
|
||||
: The non-empty Unicode code-point sequence emitted from a tag source spelling, for example `work/notes` from `#work/notes`.
|
||||
|
||||
**Tag candidate**
|
||||
: An introducer followed by a source spelling that matches the lexical grammar entirely within one literal-source run, before that run's Markdown context
|
||||
is checked for eligibility.
|
||||
|
||||
**Tag occurrence**
|
||||
: A tag candidate whose complete recognized source span is in an eligible literal-source run.
|
||||
|
||||
**Literal-source run**
|
||||
: A contiguous range of original Markdown source exposed by parsing as literal characters with no intervening Markdown escape, character reference, syntax
|
||||
token, or node boundary. Each run retains its enclosing Markdown context for eligibility checks.
|
||||
|
||||
**Eligible text**
|
||||
: A literal-source run whose enclosing context is classified as textual content by GFM 0.29-gfm or explicitly exposed as ordinary text by a Memos
|
||||
Markdown extension.
|
||||
|
||||
**Opaque Markdown node**
|
||||
: A GFM syntax node or Memos extension node whose source is not eligible for tag recognition. New extension node types are opaque by default.
|
||||
|
||||
**Source spelling**
|
||||
: The exact source substring consumed after the introducer, including ignored default-ignorable code points and ignored leading combining marks. Inline
|
||||
rendering and source-preserving operations use this spelling.
|
||||
|
||||
**Recognized source span**
|
||||
: The exact contiguous substring of original Markdown formed by the introducer and its complete source spelling.
|
||||
|
||||
**Direct tag value**
|
||||
: The identifier value emitted from a tag occurrence before hierarchy expansion. It may differ from the source spelling because default-ignorable code
|
||||
points outside a matched fully-qualified emoji sequence and ignored leading combining marks before the starter of each segment are consumed but omitted.
|
||||
|
||||
**Implied ancestor tag**
|
||||
: A slash-delimited prefix derived from a direct tag value. For example, `book` is an implied ancestor of the direct value `book/fiction`.
|
||||
|
||||
**Tag segment**
|
||||
: A non-empty component of a hierarchical tag identifier. Slashes separate segments and are not part of any segment.
|
||||
|
||||
**Display value**
|
||||
: The direct or implied value presented as a derived tag label. It preserves emitted code points exactly but does not contain ignored default-ignorable code
|
||||
points or ignored leading combining marks.
|
||||
|
||||
**Comparison key**
|
||||
: The value used to compare two display values for tag identity. It is the exact display value without case folding or Unicode
|
||||
normalization.
|
||||
|
||||
**Memo tag set**
|
||||
: The union of direct tag values and their implied ancestor tags exposed for a memo. Each exactly equal value appears once; any difference in the value's
|
||||
Unicode code-point sequence produces a separate membership.
|
||||
|
||||
**Tag metadata rule**
|
||||
: User configuration that selects tag values and supplies presentation or behavior metadata. Creating or changing a metadata rule does not create or rename
|
||||
a tag.
|
||||
|
||||
## Decision
|
||||
|
||||
### Domain ownership
|
||||
|
||||
Memo Markdown is the source of truth for tags. Stored payload tags, API fields, counts, navigation entries, completion candidates, and metadata matches are
|
||||
derived projections of recognized source occurrences.
|
||||
|
||||
There is no independently mutable tag resource in this domain model. In particular:
|
||||
|
||||
- A tag cannot be renamed without editing every memo source occurrence that should change.
|
||||
- Tag metadata may decorate or affect matching derived values, but it cannot change their source spelling or identity.
|
||||
- Persisted payload tags are rebuildable indexes, not authoritative tag records.
|
||||
- Import and export preserve memo Markdown; they do not substitute a separate canonical tag label for the spelling in source.
|
||||
|
||||
### Lexical syntax
|
||||
|
||||
Memos tag syntax is based on Unicode UAX #31 and Unicode Emoji data, with custom rules for Memos. It is not an unchanged implementation of UAX31-R8 or
|
||||
the UAX #31 Emoji Profile.
|
||||
|
||||
The normative grammar is:
|
||||
|
||||
```text
|
||||
TagCandidate := Introducer TagSourceSpelling
|
||||
Introducer := U+0023 NUMBER SIGN ("#") outside a FullyQualifiedEmoji
|
||||
TagSourceSpelling := TagSegmentSpelling ("/" TagSegmentSpelling)*
|
||||
TagSegmentSpelling := IgnoredPrefix* SegmentStarter SegmentContinuation*
|
||||
IgnoredPrefix := IgnoredDefaultIgnorable | IgnoredLeadingCombiningMark
|
||||
SegmentContinuation := ValueUnit | IgnoredDefaultIgnorable
|
||||
|
||||
SegmentStarter := EmittedXIDContinueCodePointExceptCombiningMark
|
||||
| FullyQualifiedEmoji
|
||||
| "-"
|
||||
| "+"
|
||||
| "&"
|
||||
|
||||
ValueUnit := EmittedXIDContinueCodePoint
|
||||
| FullyQualifiedEmoji
|
||||
| "-"
|
||||
| "+"
|
||||
| "&"
|
||||
|
||||
IgnoredDefaultIgnorable := Default_Ignorable_Code_Point outside a FullyQualifiedEmoji
|
||||
IgnoredLeadingCombiningMark := XID_Continue with General_Category Mn or Mc,
|
||||
minus Default_Ignorable_Code_Point, before SegmentStarter
|
||||
```
|
||||
|
||||
The grammar recognizes source spelling and emits a tag identifier as follows:
|
||||
|
||||
- Each `SegmentStarter` and `ValueUnit` emits its exact source code-point sequence. A `FullyQualifiedEmoji` therefore emits its complete matched sequence.
|
||||
- Each consumed `/` emits one U+002F SOLIDUS into the identifier.
|
||||
- `Introducer`, `IgnoredDefaultIgnorable`, and `IgnoredLeadingCombiningMark` emit nothing.
|
||||
|
||||
At every source position, the token priority is the longest `FullyQualifiedEmoji`, then `IgnoredDefaultIgnorable`, then `IgnoredLeadingCombiningMark`, then
|
||||
an emitted code point or Memos extension unit. This preserves default-ignorable code points inside matched fully-qualified emoji sequences while omitting
|
||||
them everywhere else.
|
||||
|
||||
The emitted tag identifier is the direct tag value.
|
||||
|
||||
Rules:
|
||||
|
||||
1. `EmittedXIDContinueCodePoint` is a Unicode code point with the `XID_Continue` property in Unicode 17.0, minus
|
||||
`Default_Ignorable_Code_Point` in Unicode 17.0.
|
||||
2. `EmittedXIDContinueCodePointExceptCombiningMark` additionally excludes General Category `Mn` and `Mc` code points.
|
||||
3. `_` is already included in `XID_Continue` and is not a separate extension.
|
||||
4. `FullyQualifiedEmoji` is a code-point sequence whose `RGI_Emoji_Qualification` value is `Fully_Qualified`, represented by status `fully-qualified` in
|
||||
the Emoji 17.0 `emoji-test.txt` data. It is matched atomically and before any shorter unit. The `Standalone_Component` value, represented by status
|
||||
`component`, is not included.
|
||||
5. The raw union of `Extended_Pictographic` and `Emoji_Component` is not used. That union admits bare `*`, isolated emoji components, invisible emoji tag
|
||||
characters, and code points reserved for future emoji.
|
||||
6. U+0023 ASCII NUMBER SIGN is the only tag introducer, but a `#` already matched inside the fully-qualified keycap sequence `#️⃣` is an emoji unit rather
|
||||
than an introducer. U+FE5F SMALL NUMBER SIGN and U+FF03 FULLWIDTH NUMBER SIGN are ordinary text.
|
||||
7. An ordinary `#` that does not begin a `FullyQualifiedEmoji` terminates the current identifier and may introduce another tag. The `#` in `#️⃣` is
|
||||
preserved as part of that atomic emoji value unit under the same longest-match rule as every other admitted emoji.
|
||||
8. Numeric-only and emoji-only identifiers are valid.
|
||||
9. `/` is a hierarchy separator and is consumed only between two non-empty segments. It cannot begin or end an identifier, and two `/` separators cannot be
|
||||
adjacent.
|
||||
10. `-`, `+`, and `&` are Memos-specific segment-unit extensions. They are allowed at any position, may repeat, and may form an entire segment.
|
||||
11. Default-ignorable code points inside a `FullyQualifiedEmoji` are preserved as part of that atomic value unit. Any other
|
||||
`Default_Ignorable_Code_Point` is consumed as part of the source spelling but emits no code point into the direct tag value. Ignored code points do not
|
||||
by themselves make a segment non-empty.
|
||||
12. A non-default-ignorable `XID_Continue` code point whose General Category is `Mn` (Nonspacing Mark) or `Mc` (Spacing Combining Mark) is consumed but
|
||||
omitted while it precedes the starter of its segment. The same non-default-ignorable code point is preserved as a value unit after that segment's
|
||||
starter.
|
||||
13. There is no tag-specific identifier length limit. The enclosing memo-size limit provides the resource bound.
|
||||
|
||||
The grammar is intentionally broader than a programming-language identifier grammar. It does not require the first emitted unit to be `XID_Start`:
|
||||
digits, `_`, `-`, `+`, `&`, and fully-qualified emoji can start a segment. Default-ignorable code points and non-default-ignorable `Mn` or `Mc` code points
|
||||
may occur before each segment's starter but are omitted from the value. After the starter, those non-default-ignorable `Mn` and `Mc` code points are
|
||||
preserved. A slash cannot begin a segment, and segments made only from `-`, `+`, and `&` are explicitly valid.
|
||||
|
||||
### Maximal-prefix scanning
|
||||
|
||||
After finding an introducer, the lexer consumes the longest valid `TagSourceSpelling` within the current literal-source run:
|
||||
|
||||
1. Before each segment's starter, consume default-ignorable code points and non-default-ignorable `XID_Continue` code points in General Category `Mn` or
|
||||
`Mc` into the source spelling without emitting them.
|
||||
2. Emit a segment starter, trying the longest matching `FullyQualifiedEmoji` before any shorter unit.
|
||||
3. After the starter, preserve non-default-ignorable `XID_Continue` combining marks as ordinary value units; continue consuming default-ignorable code
|
||||
points without emitting them.
|
||||
4. Consume `/` only when the following source, after any ignored prefix, can emit a segment starter; then consume that segment.
|
||||
5. Stop before a `/` that is leading, trailing, or followed only by an ignored prefix and then another `/`, a non-starter, or the end of the literal-source
|
||||
run, leaving that slash and the remaining source unconsumed.
|
||||
6. Otherwise stop before the first code point that cannot begin a valid continuation unit.
|
||||
7. Keep the valid prefix already consumed; a later invalid character does not invalidate it.
|
||||
8. Produce no candidate if the first `TagSegmentSpelling` cannot match.
|
||||
|
||||
### Candidate enumeration
|
||||
|
||||
Candidate discovery and identifier scanning use the same token priority:
|
||||
|
||||
1. Match the longest `FullyQualifiedEmoji` at the current position before interpreting any constituent code point separately. Outside an active tag
|
||||
candidate, advance past the complete sequence.
|
||||
2. If no emoji sequence matches and the current code point is an introducer, scan a tag candidate.
|
||||
3. After a successful candidate, continue at the first unconsumed source position. An ordinary terminating `#` is therefore reconsidered as a possible next
|
||||
introducer.
|
||||
4. After a failed introducer, advance by one code point so an adjacent `#` is still considered.
|
||||
5. Otherwise, advance by one code point.
|
||||
|
||||
This produces:
|
||||
|
||||
```text
|
||||
#️⃣ -> no tag; the source is one emoji
|
||||
##️⃣ -> #️⃣
|
||||
#first#️⃣ -> first#️⃣
|
||||
#first#second -> first, second
|
||||
##tag -> tag
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
| Source | Identifier | Remaining source |
|
||||
| --- | --- | --- |
|
||||
| `#foo,bar` | `foo` | `,bar` |
|
||||
| `#foo.bar` | `foo` | `.bar` |
|
||||
| `#foo:bar` | `foo` | `:bar` |
|
||||
| `#x=y` | `x` | `=y` |
|
||||
| `#price€` | `price` | `€` |
|
||||
| `#C++` | `C++` | empty |
|
||||
| `#R&D` | `R&D` | empty |
|
||||
| `#-foo` | `-foo` | empty |
|
||||
| `#foo-` | `foo-` | empty |
|
||||
| `#---` | `---` | empty |
|
||||
| `#&&` | `&&` | empty |
|
||||
| `#AB` | `AB` | empty |
|
||||
| `#foo` | `foo` | empty |
|
||||
| `#AB` | `AB` | empty |
|
||||
| `#foo` | `foo` | empty |
|
||||
| `#A️B` | `AB` | empty |
|
||||
| `#́foo` | `foo` | empty |
|
||||
| `#́` | none | U+0301 COMBINING ACUTE ACCENT |
|
||||
| `#café` | `café` | empty |
|
||||
| `#work/notes` | `work/notes` | empty |
|
||||
| `#foo/́bar` | `foo/bar` | empty |
|
||||
| `#book/` | `book` | `/` |
|
||||
| `#/book` | none | `/book` |
|
||||
| `#book//fiction` | `book` | `//fiction` |
|
||||
| `#book/fiction/` | `book/fiction` | `/` |
|
||||
| `#foo\+bar` | `foo` | `\+bar` |
|
||||
| `#R&D` | `R` | `&D` |
|
||||
|
||||
For the source below, the identifier is `foo`; the inline-code span begins immediately afterward:
|
||||
|
||||
```markdown
|
||||
#foo`test`
|
||||
```
|
||||
|
||||
### Introducer boundary
|
||||
|
||||
There is no general character boundary before the introducer. The following all contain the identifier `tag`:
|
||||
|
||||
```text
|
||||
#tag
|
||||
hello#tag
|
||||
中文#tag
|
||||
```
|
||||
|
||||
This intentionally differs from the flowing-text boundary recommendation in UAX #31. Memos can distinguish URL fragments using its Markdown parser and
|
||||
does not need to approximate URLs by forbidding a letter or number before `#`.
|
||||
|
||||
An ordinary `#` is a hard boundary between identifiers:
|
||||
|
||||
```text
|
||||
#first#second -> first, second
|
||||
##tag -> tag, introduced by the second #
|
||||
## tag -> no tag; this is an ATX heading marker followed by text
|
||||
```
|
||||
|
||||
### Markdown-context validation
|
||||
|
||||
Markdown parsing first divides the original source into literal-source runs and records each run's enclosing context. Lexical scanning then finds candidates
|
||||
within, but never across, those runs. A candidate becomes a tag occurrence only when its complete source span is in eligible text under GFM 0.29-gfm plus
|
||||
the declared Memos extensions.
|
||||
|
||||
The lexer reads literal characters from each literal-source run's original source range. It does not scan a decoded or rendered text-node value. A Markdown
|
||||
escape, character reference, or other syntax node is a hard boundary that a candidate cannot cross.
|
||||
|
||||
Tag occurrences are allowed in text contained by formatting and block structure, including emphasis, strong emphasis, strikethrough, headings, lists,
|
||||
quotes, and table cells.
|
||||
|
||||
Tag occurrences do not arise from source that is:
|
||||
|
||||
- Escaped by Markdown, as in `\#tag`.
|
||||
- Inside inline code, fenced code, or indented code.
|
||||
- Anywhere inside a Markdown link, autolink, GFM literal URL, link reference, image, or image alt text.
|
||||
- Inside raw HTML syntax or an opaque raw HTML block.
|
||||
- Inside an opaque Memos extension node, including inline or block math.
|
||||
|
||||
Consequently, a character reference that renders as `#` is not an introducer, and syntax that renders as an otherwise valid tag unit does not join the
|
||||
literal source on either side:
|
||||
|
||||
```text
|
||||
#tag -> no tag
|
||||
#tag -> no tag
|
||||
#foo\+bar -> foo
|
||||
#R&D -> R
|
||||
#R&D -> R&D
|
||||
```
|
||||
|
||||
The complete link-like node is excluded, not only its destination. Therefore neither the label nor destination produces a tag:
|
||||
|
||||
```markdown
|
||||
[release #notes](https://example.com/releases#notes)
|
||||
```
|
||||
|
||||
Context is determined by the Markdown syntax tree or AST, not by a second URL regular expression. This has two important consequences:
|
||||
|
||||
```text
|
||||
https://example.com/path#tag -> no tag when parsed as a URL/autolink
|
||||
/path#tag -> tag when the text is not parsed as a link
|
||||
```
|
||||
|
||||
Text between inline HTML tags may still contain a tag when the Markdown parser exposes that text as an ordinary text node. The HTML markup itself and
|
||||
opaque HTML blocks are never scanned.
|
||||
|
||||
### Extracted value and source span
|
||||
|
||||
- The extracted direct tag value contains the emitted identifier only, never the introducer.
|
||||
- The recognized source span contains the introducer and the entire consumed source spelling.
|
||||
- Highlighting and inline rendered tag markup use that full source span and preserve its source spelling; derived tag labels use the display value.
|
||||
- Completion replaces the consumed source spelling after the introducer; it does not replace preceding prose.
|
||||
- The original source spelling is preserved. Case folding and normalization must not occur before lexical boundaries are found.
|
||||
- A default-ignorable code point outside a matched fully-qualified emoji sequence or an ignored leading combining mark before any segment's starter belongs
|
||||
to the recognized source span but is omitted from the extracted value.
|
||||
|
||||
For example, `##tag` extracts the value `tag`, while the recognized source span is the second `#tag`.
|
||||
|
||||
For `#AB`, the recognized source span contains `#`, `A`, U+200D ZWJ, and `B`, while the extracted direct value is the two-code-point string `AB`.
|
||||
|
||||
For `#́foo`, the recognized source span includes the leading U+0301 COMBINING ACUTE ACCENT, while the extracted direct value is `foo`. In `#café`, the
|
||||
same U+0301 follows a segment starter and remains part of the extracted value.
|
||||
|
||||
### Identifier equality
|
||||
|
||||
The comparison key is the emitted display value. Two identifiers are equal only when their emitted Unicode code-point sequences are identical. Source
|
||||
spellings that differ only by ignored default-ignorable code points or ignored leading combining marks therefore compare equal.
|
||||
|
||||
Memos does not apply case folding, canonical normalization, compatibility normalization, width folding, locale-sensitive comparison, or accent folding when
|
||||
deduplicating, counting, filtering, navigating, or performing exact metadata lookup. Consequently, each pair below contains two distinct tags:
|
||||
|
||||
```text
|
||||
#Work / #work
|
||||
#café / #café
|
||||
#A / #A
|
||||
#straße / #STRASSE
|
||||
```
|
||||
|
||||
Multiple occurrences that emit exactly equal direct tag values in one memo produce one memo-tag membership. A tag metadata rule may deliberately match
|
||||
several distinct tags, but that rule does not change their identity.
|
||||
|
||||
Lexical emission applies only when recognizing Markdown source. An operand supplied to an exact filter or metadata lookup is already a tag value: it is
|
||||
compared as supplied and is not re-lexed or stripped of code points.
|
||||
|
||||
For example, `#AB`, `#AB`, and `#AB` all emit the comparison key `AB`; `#foo` and `#́foo` both emit `foo`. Each group contributes one tag membership if its
|
||||
spellings occur in the same memo. Their Markdown source remains unchanged.
|
||||
|
||||
### Hierarchy and memo membership
|
||||
|
||||
The slash `/` is a structural hierarchy separator, not an opaque character in a tag value. A direct tag value contributes itself and every slash-delimited
|
||||
ancestor prefix to the memo tag set:
|
||||
|
||||
```text
|
||||
Source occurrence: #book/fiction/history
|
||||
Direct tag value: book/fiction/history
|
||||
Memo tag set: book, book/fiction, book/fiction/history
|
||||
```
|
||||
|
||||
Hierarchy expansion has the following domain semantics:
|
||||
|
||||
- `Memo.tags` exposes both direct values and implied ancestors.
|
||||
- Exact membership filtering for an ancestor matches a memo containing only a descendant occurrence.
|
||||
- A tag count is the number of memo tag sets containing that value. One memo contributes at most one count to each direct or implied tag.
|
||||
- Rendering and export preserve the one source occurrence; hierarchy expansion does not insert ancestor hashtags into Markdown.
|
||||
|
||||
Every hierarchy segment is non-empty. A leading slash produces no candidate; a trailing or repeated slash terminates the identifier before that slash under
|
||||
the maximal-prefix rule.
|
||||
|
||||
### Pinned Unicode and emoji data
|
||||
|
||||
Tag recognition is pinned to Unicode 17.0 and Emoji 17.0. Go, browser, and Node runtime tables are not implicitly normative.
|
||||
|
||||
The pinned Unicode property assignments and Emoji data are normative, rather than the tables supplied by a particular runtime. Updating either data set
|
||||
requires an explicit specification change because newly assigned `XID_Continue` characters and newly fully-qualified emoji sequences change what source
|
||||
text means. Effects on existing data after such an update are outside scope.
|
||||
|
||||
## Conformance examples
|
||||
|
||||
The following examples are normative for the lexical and context decisions already made:
|
||||
|
||||
| Source | Extracted identifiers | Reason |
|
||||
| --- | --- | --- |
|
||||
| `#tag` | `tag` | Basic identifier |
|
||||
| `hello#tag` | `tag` | No general left boundary |
|
||||
| `#标签` | `标签` | Multilingual XID characters |
|
||||
| `#2026` | `2026` | Numeric-only identifiers are valid |
|
||||
| `#C++` | `C++` | Explicit `+` extension |
|
||||
| `#R&D` | `R&D` | Explicit `&` extension |
|
||||
| `#` followed by 101 `a` code points | all 101 `a` code points | There is no tag-specific length limit |
|
||||
| `#-foo` | `-foo` | Visible connector extensions may begin a segment |
|
||||
| `#foo-` | `foo-` | Visible connector extensions may end a segment |
|
||||
| `#---` | `---` | A segment may contain only visible connector extensions |
|
||||
| `#&&` | `&&` | Visible connector extensions may repeat |
|
||||
| `#AB` | `AB` | A non-emoji ZWJ is consumed but omitted from the value |
|
||||
| `#foo` | `foo` | An ignored leading ZWJ does not prevent a later value unit from starting the segment |
|
||||
| `#AB` | `AB` | A non-emoji ZWNJ is consumed but omitted from the value |
|
||||
| `#foo` | `foo` | An ignored leading ZWNJ does not prevent a later value unit from starting the segment |
|
||||
| `#` | none | Ignored default-ignorable code points alone do not form a segment |
|
||||
| `#A️B` | `AB` | A default-ignorable variation selector outside a matched fully-qualified emoji is omitted even after a starter |
|
||||
| `#́foo` | `foo` | A non-default-ignorable combining mark before the segment starter is consumed but omitted |
|
||||
| `#́` | none | Leading combining marks alone do not form a segment |
|
||||
| `#café` | `café` | A non-default-ignorable combining mark after the segment starter is preserved |
|
||||
| `#work/notes` | `work/notes` | Direct hierarchical identifier; memo membership also includes `work` |
|
||||
| `#foo/́bar` | `foo/bar` | Ignored leading combining-mark handling restarts for each segment |
|
||||
| `#book/` | `book` | A trailing slash is not consumed |
|
||||
| `#/book` | none | A hierarchy cannot begin with an empty segment |
|
||||
| `#book//fiction` | `book` | A repeated slash terminates before the first slash |
|
||||
| `#book/fiction/` | `book/fiction` | The valid hierarchical prefix is retained |
|
||||
| `#l·l` | `l·l` | Middle dot is `XID_Continue` |
|
||||
| `#foo‿bar` | `foo‿bar` | Connector punctuation is `XID_Continue` |
|
||||
| `#*️⃣` | `*️⃣` | Fully-qualified keycap sequence |
|
||||
| `#‼️` | `‼️` | Fully-qualified emoji sequence |
|
||||
| `#♥` | none | Bare text-presentation symbol is neither XID nor fully-qualified emoji |
|
||||
| `#♥️` | `♥️` | Fully-qualified emoji sequence |
|
||||
| `#🏻` | none | A standalone emoji component is excluded from `FullyQualifiedEmoji` |
|
||||
| `#foo,` | `foo` | Comma terminates the identifier |
|
||||
| `#price€` | `price` | Currency symbol terminates the identifier |
|
||||
| `#€budget` | none | The first body character is not a unit |
|
||||
| `#v²` | `v` | `Other_Number` is not generally `XID_Continue` |
|
||||
| `#first#second` | `first`, `second` | Ordinary `#` terminates and starts candidates |
|
||||
| `#️⃣` | none | The complete source is a fully-qualified emoji, not an introducer plus identifier |
|
||||
| `##️⃣` | `#️⃣` | The first `#` introduces the atomic keycap emoji value |
|
||||
| `#first#️⃣` | `first#️⃣` | The keycap emoji is an ordinary fully-qualified continuation unit |
|
||||
| `#tag` | none | Fullwidth number sign is not an introducer |
|
||||
| `﹟tag` | none | Small number sign is not an introducer |
|
||||
| `#tag` | none | A character reference is not a literal introducer |
|
||||
| `#tag` | none | Decoded text is not rescanned for candidates |
|
||||
| `\#tag` | none | Escaped introducer |
|
||||
| `#foo\+bar` | `foo` | A Markdown escape terminates the literal-source range |
|
||||
| `#R&D` | `R` | A character reference terminates the literal-source range |
|
||||
| `#R&D` | `R&D` | A literal ampersand is a value unit |
|
||||
| `` `#tag` `` | none | Inline-code context |
|
||||
| `**#urgent**` | `urgent` | Formatted normal text remains eligible |
|
||||
| `https://example.com/#tag` | none | GFM URL/autolink context |
|
||||
| `/path#tag` | `tag` | Plain text path is not guessed to be a URL |
|
||||
| `[hello#tag](https://example.com)` | none | The complete link is excluded |
|
||||
| `$#tag$` | none | A Memos math node is opaque |
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Tag meaning is specified independently of any single parser or issue.
|
||||
- Common multilingual tags, numeric tags, hierarchy characters, and emoji remain expressive.
|
||||
- Hierarchical ancestors have one consistent meaning across API membership, exact filters, navigation, and counts.
|
||||
- `hello#tag` works consistently while actual Markdown links and URLs are excluded structurally.
|
||||
- Markdown punctuation and delimiters such as comma, backtick, currency, and mathematical operators stop an identifier predictably.
|
||||
- Exact equality preserves every emitted code-point distinction and avoids locale-dependent identity rules.
|
||||
- Ignoring default-ignorable code points outside emoji avoids invisible tag distinctions while preserving matched fully-qualified emoji sequences.
|
||||
- Ignoring leading non-default-ignorable `Mn` and `Mc` code points only before each segment's visible starter avoids invisible-leading segments without
|
||||
breaking decomposed writing inside words.
|
||||
- Keeping `-`, `+`, and `&` position-independent avoids a second connector-validation layer.
|
||||
- The memo-size limit provides the only length bound, avoiding tag-specific counting and overflow rules.
|
||||
- All consumers can implement the same language contract.
|
||||
- Unicode repertoire changes require an explicit specification update rather than depending on runtime tables.
|
||||
|
||||
### Negative
|
||||
|
||||
- Fully-qualified emoji matching requires sequence-aware data, not a simple code-point character class.
|
||||
- Pinning Unicode data requires maintenance when Unicode and Emoji data are updated.
|
||||
- Visually indistinguishable or canonically equivalent source spellings whose emitted values differ remain separate tags unless the user edits their memo
|
||||
sources to make those values identical.
|
||||
- Source spellings that differ only by ignored default-ignorable code points intentionally collapse to the same emitted tag value.
|
||||
- Source spellings that differ only by ignored leading combining marks intentionally collapse to the same emitted tag value.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### Keep `L/M/N/S` and patch URL detection
|
||||
|
||||
Rejected. `\p{S}` admits currency, mathematical operators, modifier symbols, and Markdown backticks merely as a side effect of supporting emoji. URL detection
|
||||
would still differ across consumers and would duplicate the Markdown parser.
|
||||
|
||||
### Require a whitespace or punctuation boundary before `#`
|
||||
|
||||
Rejected. It makes the intentionally supported `hello#tag` form invalid and uses a lexical approximation to solve a structural URL problem.
|
||||
|
||||
### Adopt UAX31-R8 unchanged
|
||||
|
||||
Rejected. UAX31-R8 recommends a flowing-text left boundary that does not match the product decision, and its raw emoji component repertoire is broader than
|
||||
the desired complete-sequence behavior. Memos instead defines an explicit UAX #31 profile.
|
||||
|
||||
### Use `Extended_Pictographic | Emoji_Component` as code-point characters
|
||||
|
||||
Rejected. It admits bare `*`, isolated modifiers and regional indicators, invisible emoji tag characters, and unassigned code points reserved for future
|
||||
emoji. Matching only fully-qualified entries expresses the intended user-visible unit.
|
||||
|
||||
### Keep the 100-code-point limit
|
||||
|
||||
Rejected. The value is arbitrary, counts code points rather than user-perceived characters, and currently has three different overflow behaviors. Memo-size
|
||||
limits already bound work.
|
||||
|
||||
### Reject an entire candidate when a later invalid character appears
|
||||
|
||||
Rejected. Hashtags in flowing text conventionally end at punctuation. Maximal-prefix behavior makes `#foo,` and `#foo` followed by inline code predictable.
|
||||
|
||||
### Allow empty hierarchy segments
|
||||
|
||||
Rejected. Empty segments have no useful navigation or membership meaning and would require special handling in hierarchy expansion. The grammar permits `/`
|
||||
only between two non-empty segments.
|
||||
|
||||
### Reject a whole hierarchy after an invalid slash
|
||||
|
||||
Rejected. Keeping the longest valid prefix makes a trailing or repeated slash behave like other terminating syntax and avoids erasing an otherwise valid
|
||||
tag. The slash and remaining source stay ordinary Markdown.
|
||||
|
||||
### Restrict visible connectors by position
|
||||
|
||||
Rejected. Requiring `-`, `+`, or `&` to be medial, non-repeating, or accompanied by a letter would add validation rules without resolving a structural
|
||||
ambiguity. They remain ordinary segment units; only `/` has structural meaning.
|
||||
|
||||
### Recognize compatibility number signs
|
||||
|
||||
Rejected. U+FE5F SMALL NUMBER SIGN and U+FF03 FULLWIDTH NUMBER SIGN are visually similar to `#` but are not Markdown syntax. Recognizing only ASCII U+0023
|
||||
keeps the introducer rule literal and unambiguous.
|
||||
|
||||
### Special-case the number-sign keycap
|
||||
|
||||
Rejected. `#️⃣` follows the same longest-sequence matching rule as every other admitted fully-qualified emoji. Excluding it or splitting its code points
|
||||
would add an exception to the emoji profile.
|
||||
|
||||
### Scan decoded Markdown text
|
||||
|
||||
Rejected. Decoding character references and escapes before tag scanning would allow non-literal introducers and require source reconstruction for exact
|
||||
spans. Tag lexing uses original source ranges; Markdown syntax nodes act as boundaries.
|
||||
|
||||
### Leave the Markdown dialect parser-defined
|
||||
|
||||
Rejected. Parser-dependent context would allow identical source to produce different tags. GFM 0.29-gfm is the structural baseline; additional Memos
|
||||
extensions are opaque unless explicitly declared transparent to tag recognition.
|
||||
|
||||
## References
|
||||
|
||||
- [Unicode 17.0 Standard Annex #31: Unicode Identifiers and Syntax](https://www.unicode.org/reports/tr31/tr31-43.html)
|
||||
- [UAX #31 Hashtag Identifiers](https://www.unicode.org/reports/tr31/tr31-43.html#Hashtag_Identifiers)
|
||||
- [UAX #31 Emoji Profile](https://www.unicode.org/reports/tr31/tr31-43.html#Emoji_Profile)
|
||||
- [Unicode Emoji 17.0 (UTS #51)](https://www.unicode.org/reports/tr51/tr51-29.html)
|
||||
- [Unicode Emoji 17.0 test data](https://www.unicode.org/Public/17.0.0/emoji/emoji-test.txt)
|
||||
- [GitHub Flavored Markdown specification](https://github.github.com/gfm/)
|
||||
- [Memos tag documentation](https://usememos.com/docs/usage/tags)
|
||||
60
docs/adr/README.md
Normal file
60
docs/adr/README.md
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
# Architecture Decision Records
|
||||
|
||||
Architecture Decision Records (ADRs) document significant technical and product decisions whose rationale should remain available after the related
|
||||
implementation work is complete.
|
||||
|
||||
## Index
|
||||
|
||||
| ADR | Title | Status | Date |
|
||||
| --- | --- | --- | --- |
|
||||
| [0001](0001-tag-syntax-and-recognition.md) | Tag Syntax and Recognition | Accepted | 2026-08-01 |
|
||||
|
||||
## Conventions
|
||||
|
||||
- Name files `NNNN-short-kebab-case-title.md`, using a four-digit number that is never reused.
|
||||
- Assign the next number when an ADR is opened, even if an earlier ADR is later rejected or superseded.
|
||||
- Use the title `# ADR NNNN: Title` and record dates as `YYYY-MM-DD`.
|
||||
- Keep accepted ADRs as historical records. Replace a decision with a new ADR rather than rewriting the original rationale.
|
||||
- When one ADR replaces another, add `Supersedes: ADR NNNN` to the new ADR and `Superseded by: ADR NNNN` to the old ADR.
|
||||
|
||||
## Statuses
|
||||
|
||||
- **Proposed**: Under discussion; unresolved questions may remain.
|
||||
- **Accepted**: Approved as the decision to implement and maintain.
|
||||
- **Rejected**: Considered but not selected.
|
||||
- **Superseded**: Replaced by a later ADR.
|
||||
|
||||
## Suggested structure
|
||||
|
||||
Each ADR should contain enough information to understand the decision without reconstructing the original discussion:
|
||||
|
||||
```markdown
|
||||
# ADR NNNN: Title
|
||||
|
||||
Status: Proposed
|
||||
|
||||
Date: YYYY-MM-DD
|
||||
|
||||
## Context
|
||||
|
||||
## Decision drivers
|
||||
|
||||
## Decision
|
||||
|
||||
## Consequences
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
## Open questions before acceptance
|
||||
|
||||
## References
|
||||
```
|
||||
|
||||
Optional sections may be omitted when they do not add useful context.
|
||||
|
||||
## Process
|
||||
|
||||
1. Create a `Proposed` ADR with the next available number and add it to the index.
|
||||
2. Discuss the proposal with maintainers, updating the ADR as the decision develops.
|
||||
3. Change the status to `Accepted` or `Rejected` when the outcome is clear.
|
||||
4. If an accepted decision changes materially, create a new ADR and cross-link the superseding records.
|
||||
81
docs/glossary.md
Normal file
81
docs/glossary.md
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
# Memos Domain Glossary
|
||||
|
||||
This glossary defines product and domain language shared across Memos design documents. Protocol, Unicode, and parser-specific terms remain in the ADRs
|
||||
that use them.
|
||||
|
||||
## Tags
|
||||
|
||||
### Tag
|
||||
|
||||
A classification value in a memo tag set, derived from one or more tag occurrences either as a direct tag value or as an implied ancestor. A tag is not a
|
||||
durable entity and cannot be independently created or renamed. Changing a tag's spelling requires editing the relevant memo source occurrences.
|
||||
|
||||
### Tag introducer
|
||||
|
||||
An ASCII `#` character (U+0023) that is not part of a matched fully-qualified emoji sequence and starts a tag candidate. Visually similar fullwidth and small
|
||||
number-sign characters are not introducers.
|
||||
|
||||
### Tag occurrence
|
||||
|
||||
An eligible source span in memo Markdown formed by a tag introducer and its consumed source spelling. The spelling emits a tag identifier, which is the
|
||||
direct tag value. Recognition rules, including excluded Markdown contexts, are defined by the active tag-syntax profile.
|
||||
|
||||
### Eligible tag text
|
||||
|
||||
A contiguous original-source range with no intervening Markdown syntax boundary, classified as textual content by GFM 0.29-gfm or explicitly exposed as
|
||||
ordinary text by a Memos Markdown extension. Extension nodes are opaque by default, so their source does not produce tags unless the extension definition
|
||||
opts in.
|
||||
|
||||
### Source spelling
|
||||
|
||||
The exact source substring consumed after the `#` introducer. It includes default-ignorable code points outside matched fully-qualified emoji sequences and
|
||||
ignored leading combining marks before each segment's starter, both of which the lexer ignores. Inline rendering and source-preserving operations use this
|
||||
spelling.
|
||||
|
||||
### Direct tag value
|
||||
|
||||
The identifier emitted from one tag occurrence before hierarchy expansion. `book/fiction` is the direct value produced by `#book/fiction`.
|
||||
Default-ignorable code points outside matched fully-qualified emoji sequences and ignored leading combining marks before each segment's starter are
|
||||
consumed from the source spelling but omitted from this value.
|
||||
|
||||
### Implied ancestor tag
|
||||
|
||||
A slash-delimited prefix derived from a direct tag value. A direct value of `book/fiction/history` implies the ancestor tags `book` and `book/fiction`.
|
||||
|
||||
### Tag segment
|
||||
|
||||
A non-empty component of a hierarchical tag identifier. `/` separates segments and is consumed only when another non-empty segment follows. A leading slash
|
||||
produces no tag; a trailing or repeated slash terminates the identifier before that slash. `-`, `+`, and `&` are ordinary segment units: they may appear at
|
||||
any position, repeat, or form a complete segment. Default-ignorable code points and ignored leading combining marks do not make a segment non-empty.
|
||||
|
||||
### Tag identifier
|
||||
|
||||
The non-empty Unicode code-point sequence emitted from a tag source spelling. The identifier excludes the introducer and all ignored source code points.
|
||||
|
||||
### Display value
|
||||
|
||||
The direct or implied value presented as a derived tag label. Memos does not normalize or case-fold emitted code points for identity, but ignored
|
||||
default-ignorable code points and ignored leading combining marks are not part of the display value.
|
||||
|
||||
### Comparison key
|
||||
|
||||
The value used for deduplication, counting, filtering, navigation, and exact metadata matching. It is identical to the emitted display value. Two tags
|
||||
compare as equal only when those emitted Unicode code-point sequences are identical; case and canonical or compatibility-equivalent spellings remain
|
||||
distinct. Source spellings that differ only by ignored default-ignorable code points or ignored leading combining marks compare equal. Inputs to exact
|
||||
value filters and metadata lookups are already tag values and are compared as supplied; they are not re-lexed as source spellings.
|
||||
|
||||
### Memo tag set
|
||||
|
||||
The union of direct tag values and their implied ancestor tags for one memo, exposed as `Memo.tags`. It is rebuildable from the memo's Markdown and is not an
|
||||
authoritative source of tags. Each exactly equal emitted value appears once, even when produced both directly and as an ancestor. Different source
|
||||
spellings remain separate only when they emit different code-point sequences.
|
||||
|
||||
### Tag metadata rule
|
||||
|
||||
User configuration that selects tag values and supplies presentation or behavior metadata, such as color or content blurring. A metadata rule may match
|
||||
multiple values and does not create, own, or rename a tag.
|
||||
|
||||
### Tag count
|
||||
|
||||
The number of memo tag sets containing an exactly equal direct or implied tag value, not the number of textual occurrences. A memo containing only
|
||||
`#book/fiction` contributes one to both the `book` and `book/fiction` counts.
|
||||
|
|
@ -54,9 +54,11 @@ stmt, _ := engine.CompileToStatement(ctx, `has_task_list && visibility == "PUBLI
|
|||
`timestamp("2006-01-02T15:04:05Z")` / `timestamp(<epoch-seconds>)`. These fold
|
||||
to epoch seconds at compile time — `now` is frozen once per compile (injectable
|
||||
for tests via the engine clock) — so the backing columns stay unchanged.
|
||||
- **Tag Operations** — `tag in [...]` and `"tag" in tags` become JSON array
|
||||
predicates. SQLite uses `LIKE` patterns, MySQL uses `JSON_CONTAINS`, and
|
||||
Postgres uses `@>`.
|
||||
- **Tag Operations** — `tag in [...]`, `"tag" in tags`, and tag comprehension
|
||||
predicates evaluate individual JSON array elements. Equality and string
|
||||
matching are case-sensitive, without Unicode normalization; `%` and `_` in
|
||||
string operands are literal. Hierarchy works through implied ancestors already
|
||||
present in the memo tag set, not through prefix matching.
|
||||
- **Boolean Flags** — Fields such as `has_task_list` render as `IS TRUE` equality
|
||||
checks, or comparisons against `CAST('true' AS JSON)` depending on the dialect.
|
||||
- **String Matching** — `content.contains(x)`, `content.startsWith(x)`, and
|
||||
|
|
|
|||
|
|
@ -2,9 +2,12 @@ package filter
|
|||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
func TestCompileAcceptsStandardTagEqualityPredicate(t *testing.T) {
|
||||
|
|
@ -85,6 +88,275 @@ func TestCompileContainsEscapesLikeWildcards(t *testing.T) {
|
|||
require.Equal(t, []any{`%50\%\_off%`}, stmt.Args)
|
||||
}
|
||||
|
||||
func TestRenderTagMembershipIsExactPerDialect(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
engine, err := NewEngine(NewSchema())
|
||||
require.NoError(t, err)
|
||||
|
||||
const tag = `work_%"quoted"`
|
||||
cases := []struct {
|
||||
dialect DialectName
|
||||
fragments []string
|
||||
}{
|
||||
{DialectSQLite, []string{"json_each(", "COLLATE BINARY"}},
|
||||
{DialectMySQL, []string{"JSON_TABLE(", "CAST(tag_item.value AS BINARY)"}},
|
||||
{DialectPostgres, []string{"jsonb_array_elements_text(", `(tag_item.value COLLATE "C")`}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
stmt, err := engine.CompileToStatement(context.Background(), `tag in ["work_%\"quoted\""]`, RenderOptions{Dialect: tc.dialect})
|
||||
require.NoError(t, err, tc.dialect)
|
||||
for _, fragment := range tc.fragments {
|
||||
require.Contains(t, stmt.SQL, fragment, tc.dialect)
|
||||
}
|
||||
require.NotContains(t, stmt.SQL, " LIKE ", tc.dialect)
|
||||
require.Equal(t, []any{tag}, stmt.Args, tc.dialect)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderTagExistsEqualityIsExactPerDialect(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
engine, err := NewEngine(NewSchema())
|
||||
require.NoError(t, err)
|
||||
|
||||
const tag = `work_%"quoted"`
|
||||
cases := []struct {
|
||||
dialect DialectName
|
||||
fragments []string
|
||||
}{
|
||||
{DialectSQLite, []string{"json_each(", "COLLATE BINARY"}},
|
||||
{DialectMySQL, []string{"JSON_TABLE(", "CAST(tag_item.value AS BINARY)"}},
|
||||
{DialectPostgres, []string{"jsonb_array_elements_text(", `(tag_item.value COLLATE "C")`}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
stmt, err := engine.CompileToStatement(context.Background(), `tags.exists(t, t == "work_%\"quoted\"")`, RenderOptions{Dialect: tc.dialect})
|
||||
require.NoError(t, err, tc.dialect)
|
||||
for _, fragment := range tc.fragments {
|
||||
require.Contains(t, stmt.SQL, fragment, tc.dialect)
|
||||
}
|
||||
require.NotContains(t, stmt.SQL, " LIKE ", tc.dialect)
|
||||
require.Equal(t, []any{tag}, stmt.Args, tc.dialect)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderTagComprehensionEqualityIsExactAndUnboundedOnMySQL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
engine, err := NewEngine(NewSchema())
|
||||
require.NoError(t, err)
|
||||
for _, expression := range []string{`tags.all(t, t == "Work")`, `tags.exists_one(t, t == "Work")`} {
|
||||
stmt, err := engine.CompileToStatement(context.Background(), expression, RenderOptions{Dialect: DialectMySQL})
|
||||
require.NoError(t, err)
|
||||
require.Contains(t, stmt.SQL, "value LONGTEXT PATH '$'")
|
||||
require.Contains(t, stmt.SQL, "CAST(tag_item.value AS BINARY) = CAST(? AS BINARY)")
|
||||
require.NotContains(t, stmt.SQL, "VARCHAR(512)")
|
||||
require.Equal(t, []any{"Work"}, stmt.Args)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderTagComprehensionEqualityUsesBinaryPostgresCollation(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
engine, err := NewEngine(NewSchema())
|
||||
require.NoError(t, err)
|
||||
for _, expression := range []string{`tags.all(t, t == "Work")`, `tags.exists_one(t, t == "Work")`} {
|
||||
stmt, err := engine.CompileToStatement(context.Background(), expression, RenderOptions{Dialect: DialectPostgres})
|
||||
require.NoError(t, err)
|
||||
require.Contains(t, stmt.SQL, `(tag_item.value COLLATE "C") = ($1::text COLLATE "C")`)
|
||||
require.Equal(t, []any{"Work"}, stmt.Args)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderTagStringPredicatesAreExactPerDialect(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
engine, err := NewEngine(NewSchema())
|
||||
require.NoError(t, err)
|
||||
|
||||
predicates := []struct {
|
||||
name string
|
||||
expression string
|
||||
sqliteSQL string
|
||||
sqliteArgs []any
|
||||
likePattern string
|
||||
usesLike bool
|
||||
}{
|
||||
{name: "equals", expression: `t == "Work_%!"`, sqliteSQL: "COLLATE BINARY", sqliteArgs: []any{"Work_%!"}},
|
||||
{name: "startsWith", expression: `t.startsWith("Work_%!")`, sqliteSQL: "instr(tag_item.value, ?) = 1", sqliteArgs: []any{"Work_%!"}, likePattern: "Work!_!%!!%", usesLike: true},
|
||||
{name: "endsWith", expression: `t.endsWith("Work_%!")`, sqliteSQL: "substr(tag_item.value, -length(?))", sqliteArgs: []any{"Work_%!", "Work_%!"}, likePattern: "%Work!_!%!!", usesLike: true},
|
||||
{name: "contains", expression: `t.contains("Work_%!")`, sqliteSQL: "instr(tag_item.value, ?) > 0", sqliteArgs: []any{"Work_%!"}, likePattern: "%Work!_!%!!%", usesLike: true},
|
||||
}
|
||||
|
||||
for _, comprehension := range []struct {
|
||||
kind string
|
||||
sqlFragment string
|
||||
}{
|
||||
{kind: "exists", sqlFragment: "EXISTS (SELECT 1"},
|
||||
{kind: "all", sqlFragment: "NOT EXISTS (SELECT 1"},
|
||||
{kind: "exists_one", sqlFragment: "SELECT COUNT(*)"},
|
||||
} {
|
||||
for _, predicate := range predicates {
|
||||
expression := fmt.Sprintf("tags.%s(t, %s)", comprehension.kind, predicate.expression)
|
||||
t.Run(comprehension.kind+"/"+predicate.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
sqliteStmt, err := engine.CompileToStatement(context.Background(), expression, RenderOptions{Dialect: DialectSQLite})
|
||||
require.NoError(t, err)
|
||||
require.Contains(t, sqliteStmt.SQL, "json_each(")
|
||||
require.Contains(t, sqliteStmt.SQL, comprehension.sqlFragment)
|
||||
require.Contains(t, sqliteStmt.SQL, predicate.sqliteSQL)
|
||||
require.NotContains(t, sqliteStmt.SQL, "memos_unicode_lower")
|
||||
require.Equal(t, predicate.sqliteArgs, sqliteStmt.Args)
|
||||
|
||||
mysqlStmt, err := engine.CompileToStatement(context.Background(), expression, RenderOptions{Dialect: DialectMySQL})
|
||||
require.NoError(t, err)
|
||||
require.Contains(t, mysqlStmt.SQL, "JSON_TABLE(")
|
||||
require.Contains(t, mysqlStmt.SQL, comprehension.sqlFragment)
|
||||
require.Contains(t, mysqlStmt.SQL, "CAST(tag_item.value AS BINARY)")
|
||||
|
||||
postgresStmt, err := engine.CompileToStatement(context.Background(), expression, RenderOptions{Dialect: DialectPostgres})
|
||||
require.NoError(t, err)
|
||||
require.Contains(t, postgresStmt.SQL, "jsonb_array_elements_text(")
|
||||
require.Contains(t, postgresStmt.SQL, comprehension.sqlFragment)
|
||||
require.Contains(t, postgresStmt.SQL, `(tag_item.value COLLATE "C")`)
|
||||
|
||||
if predicate.usesLike {
|
||||
require.Contains(t, mysqlStmt.SQL, " LIKE ")
|
||||
require.Contains(t, mysqlStmt.SQL, "ESCAPE '!'")
|
||||
require.Contains(t, postgresStmt.SQL, " LIKE ")
|
||||
require.Contains(t, postgresStmt.SQL, "ESCAPE '!'")
|
||||
require.Equal(t, []any{predicate.likePattern}, mysqlStmt.Args)
|
||||
require.Equal(t, []any{predicate.likePattern}, postgresStmt.Args)
|
||||
} else {
|
||||
require.NotContains(t, mysqlStmt.SQL, " LIKE ")
|
||||
require.NotContains(t, postgresStmt.SQL, " LIKE ")
|
||||
require.Equal(t, []any{"Work_%!"}, mysqlStmt.Args)
|
||||
require.Equal(t, []any{"Work_%!"}, postgresStmt.Args)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderEmptyTagListIsFalse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
engine, err := NewEngine(NewSchema())
|
||||
require.NoError(t, err)
|
||||
for _, dialect := range []DialectName{DialectSQLite, DialectMySQL, DialectPostgres} {
|
||||
stmt, err := engine.CompileToStatement(context.Background(), `tag in []`, RenderOptions{Dialect: dialect})
|
||||
require.NoError(t, err, dialect)
|
||||
require.Equal(t, "1 = 0", stmt.SQL, dialect)
|
||||
require.Empty(t, stmt.Args, dialect)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderTagStringPredicatesSQLiteBehavior(t *testing.T) {
|
||||
db, err := sql.Open("sqlite", ":memory:")
|
||||
require.NoError(t, err)
|
||||
db.SetMaxOpenConns(1)
|
||||
t.Cleanup(func() { require.NoError(t, db.Close()) })
|
||||
|
||||
_, err = db.Exec(`CREATE TABLE memo (id INTEGER PRIMARY KEY, payload TEXT)`)
|
||||
require.NoError(t, err)
|
||||
for _, fixture := range []struct {
|
||||
id int
|
||||
payload string
|
||||
}{
|
||||
{1, `{"tags":["Work_%done"]}`},
|
||||
{2, `{"tags":["work_%done"]}`},
|
||||
{3, `{"tags":["WorkX%done"]}`},
|
||||
{4, `{"tags":["Work_XXdone"]}`},
|
||||
{5, `{"tags":["Work_%done","other"]}`},
|
||||
{6, `{"tags":["Work_%done","Work_%done"]}`},
|
||||
{7, `{"tags":["preWork_%done"]}`},
|
||||
{8, `{"tags":["Work_%doneSuffix"]}`},
|
||||
} {
|
||||
_, err = db.Exec(`INSERT INTO memo (id, payload) VALUES (?, ?)`, fixture.id, fixture.payload)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
engine, err := NewEngine(NewSchema())
|
||||
require.NoError(t, err)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
predicate string
|
||||
exists []int
|
||||
all []int
|
||||
existsOne []int
|
||||
}{
|
||||
{
|
||||
name: "equality",
|
||||
predicate: `t == "Work_%done"`,
|
||||
exists: []int{1, 5, 6},
|
||||
all: []int{1, 6},
|
||||
existsOne: []int{1, 5},
|
||||
},
|
||||
{
|
||||
name: "startsWith",
|
||||
predicate: `t.startsWith("Work_%done")`,
|
||||
exists: []int{1, 5, 6, 8},
|
||||
all: []int{1, 6, 8},
|
||||
existsOne: []int{1, 5, 8},
|
||||
},
|
||||
{
|
||||
name: "endsWith",
|
||||
predicate: `t.endsWith("Work_%done")`,
|
||||
exists: []int{1, 5, 6, 7},
|
||||
all: []int{1, 6, 7},
|
||||
existsOne: []int{1, 5, 7},
|
||||
},
|
||||
{
|
||||
name: "contains",
|
||||
predicate: `t.contains("Work_%done")`,
|
||||
exists: []int{1, 5, 6, 7, 8},
|
||||
all: []int{1, 6, 7, 8},
|
||||
existsOne: []int{1, 5, 7, 8},
|
||||
},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
for _, comprehension := range []struct {
|
||||
name string
|
||||
want []int
|
||||
}{
|
||||
{name: "exists", want: tc.exists},
|
||||
{name: "all", want: tc.all},
|
||||
{name: "exists_one", want: tc.existsOne},
|
||||
} {
|
||||
t.Run(comprehension.name, func(t *testing.T) {
|
||||
stmt, err := engine.CompileToStatement(
|
||||
context.Background(),
|
||||
fmt.Sprintf("tags.%s(t, %s)", comprehension.name, tc.predicate),
|
||||
RenderOptions{Dialect: DialectSQLite},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, comprehension.want, selectMemoIDs(t, db, stmt))
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func selectMemoIDs(t *testing.T, db *sql.DB, stmt Statement) []int {
|
||||
t.Helper()
|
||||
|
||||
rows, err := db.Query(`SELECT id FROM memo WHERE `+stmt.SQL+` ORDER BY id`, stmt.Args...)
|
||||
require.NoError(t, err)
|
||||
defer rows.Close()
|
||||
|
||||
var ids []int
|
||||
for rows.Next() {
|
||||
var id int
|
||||
require.NoError(t, rows.Scan(&id))
|
||||
ids = append(ids, id)
|
||||
}
|
||||
require.NoError(t, rows.Err())
|
||||
return ids
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// Cross-dialect rendering tests (no DB required; complements the SQLite-only
|
||||
// behavioral tests in store/test by asserting MySQL/Postgres SQL generation).
|
||||
|
|
@ -158,10 +430,11 @@ func TestRenderTagsAllPerDialect(t *testing.T) {
|
|||
cases := []struct {
|
||||
dialect DialectName
|
||||
fragments []string
|
||||
args []any
|
||||
}{
|
||||
{DialectSQLite, []string{"NOT EXISTS", "json_each(", "!= '[]'", "memos_unicode_lower(value)"}},
|
||||
{DialectPostgres, []string{"NOT EXISTS", "jsonb_array_elements_text(", "jsonb_array_length(", "value ILIKE"}},
|
||||
{DialectMySQL, []string{"NOT EXISTS", "JSON_TABLE(", "JSON_LENGTH(", "value LIKE"}},
|
||||
{DialectSQLite, []string{"NOT EXISTS", "json_each(", "json_array_length(", "instr(tag_item.value, ?) = 1"}, []any{"work/"}},
|
||||
{DialectPostgres, []string{"NOT EXISTS", "jsonb_array_elements_text(", "jsonb_array_length(", `(tag_item.value COLLATE "C") LIKE`}, []any{"work/%"}},
|
||||
{DialectMySQL, []string{"NOT EXISTS", "JSON_TABLE(", "JSON_LENGTH(", "CAST(tag_item.value AS BINARY) LIKE"}, []any{"work/%"}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
stmt, err := engine.CompileToStatement(context.Background(), `tags.all(t, t.startsWith("work/"))`, RenderOptions{Dialect: tc.dialect})
|
||||
|
|
@ -169,7 +442,7 @@ func TestRenderTagsAllPerDialect(t *testing.T) {
|
|||
for _, frag := range tc.fragments {
|
||||
require.Contains(t, stmt.SQL, frag, "dialect %s", tc.dialect)
|
||||
}
|
||||
require.Equal(t, []any{"work/%"}, stmt.Args, "dialect %s", tc.dialect)
|
||||
require.Equal(t, tc.args, stmt.Args, "dialect %s", tc.dialect)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -448,30 +448,16 @@ func (r *renderer) renderTagInList(values []ValueExpr) (renderResult, error) {
|
|||
return renderResult{}, errors.New("tags must be compared with string literals")
|
||||
}
|
||||
|
||||
switch r.dialect {
|
||||
case DialectSQLite:
|
||||
// Support hierarchical tags: match exact tag OR tags with this prefix (e.g., "book" matches "book" and "book/something")
|
||||
exactMatch := fmt.Sprintf("%s LIKE %s", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`%%"%s"%%`, str)))
|
||||
prefixMatch := fmt.Sprintf("%s LIKE %s", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`%%"%s/%%`, str)))
|
||||
expr := fmt.Sprintf("(%s OR %s)", exactMatch, prefixMatch)
|
||||
conditions = append(conditions, expr)
|
||||
case DialectMySQL:
|
||||
// Support hierarchical tags: match exact tag OR tags with this prefix
|
||||
exactMatch := fmt.Sprintf("JSON_CONTAINS(%s, %s)", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`"%s"`, str)))
|
||||
prefixMatch := fmt.Sprintf("%s LIKE %s", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`%%"%s/%%`, str)))
|
||||
expr := fmt.Sprintf("(%s OR %s)", exactMatch, prefixMatch)
|
||||
conditions = append(conditions, expr)
|
||||
case DialectPostgres:
|
||||
// Support hierarchical tags: match exact tag OR tags with this prefix
|
||||
exactMatch := fmt.Sprintf("%s @> jsonb_build_array(%s::json)", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`"%s"`, str)))
|
||||
prefixMatch := fmt.Sprintf("(%s)::text LIKE %s", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`%%"%s/%%`, str)))
|
||||
expr := fmt.Sprintf("(%s OR %s)", exactMatch, prefixMatch)
|
||||
conditions = append(conditions, expr)
|
||||
default:
|
||||
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
|
||||
condition, err := r.renderJSONListContains(field, str)
|
||||
if err != nil {
|
||||
return renderResult{}, err
|
||||
}
|
||||
conditions = append(conditions, condition.sql)
|
||||
}
|
||||
|
||||
if len(conditions) == 0 {
|
||||
return renderResult{sql: "1 = 0"}, nil
|
||||
}
|
||||
if len(conditions) == 1 {
|
||||
return renderResult{sql: conditions[0]}, nil
|
||||
}
|
||||
|
|
@ -498,19 +484,11 @@ func (r *renderer) renderElementInCondition(cond *ElementInCondition) (renderRes
|
|||
return renderResult{}, errors.New("tags membership requires string literal")
|
||||
}
|
||||
|
||||
switch r.dialect {
|
||||
case DialectSQLite:
|
||||
sql := fmt.Sprintf("%s LIKE %s", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`%%"%s"%%`, str)))
|
||||
return renderResult{sql: sql}, nil
|
||||
case DialectMySQL:
|
||||
sql := fmt.Sprintf("JSON_CONTAINS(%s, %s)", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`"%s"`, str)))
|
||||
return renderResult{sql: sql}, nil
|
||||
case DialectPostgres:
|
||||
sql := fmt.Sprintf("%s @> jsonb_build_array(%s::json)", jsonArrayExpr(r.dialect, field), r.addArg(fmt.Sprintf(`"%s"`, str)))
|
||||
return renderResult{sql: sql}, nil
|
||||
default:
|
||||
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
|
||||
}
|
||||
return r.renderJSONListContains(field, str)
|
||||
}
|
||||
|
||||
func (r *renderer) renderJSONListContains(field Field, value string) (renderResult, error) {
|
||||
return r.renderTagComprehension(field, &EqualsPredicate{Value: value}, ComprehensionExists)
|
||||
}
|
||||
|
||||
func (r *renderer) renderScalarInCondition(field Field, values []ValueExpr) (renderResult, error) {
|
||||
|
|
@ -618,181 +596,116 @@ func (r *renderer) renderListComprehension(cond *ListComprehensionCondition) (re
|
|||
return renderResult{}, errors.Errorf("field %q is not a JSON list", cond.Field)
|
||||
}
|
||||
|
||||
if cond.Kind == ComprehensionAll {
|
||||
return r.renderTagAll(field, cond.Predicate)
|
||||
}
|
||||
|
||||
if cond.Kind == ComprehensionExistsOne {
|
||||
return r.renderTagExistsOne(field, cond.Predicate)
|
||||
}
|
||||
|
||||
// Render based on predicate type
|
||||
switch pred := cond.Predicate.(type) {
|
||||
case *EqualsPredicate:
|
||||
return r.renderTagEquals(field, pred.Value, cond.Kind)
|
||||
case *StartsWithPredicate:
|
||||
return r.renderTagStartsWith(field, pred.Prefix, cond.Kind)
|
||||
case *EndsWithPredicate:
|
||||
return r.renderTagEndsWith(field, pred.Suffix, cond.Kind)
|
||||
case *ContainsPredicate:
|
||||
return r.renderTagContains(field, pred.Substring, cond.Kind)
|
||||
default:
|
||||
return renderResult{}, errors.Errorf("unsupported predicate type %T in comprehension", pred)
|
||||
}
|
||||
return r.renderTagComprehension(field, cond.Predicate, cond.Kind)
|
||||
}
|
||||
|
||||
// renderTagAll renders tags.all(t, <pred>): the array is non-empty AND no element
|
||||
// fails the predicate. Element predicates use plain CEL semantics (case-insensitive
|
||||
// for startsWith/endsWith/contains, case-sensitive for ==), evaluated per element.
|
||||
func (r *renderer) renderTagAll(field Field, pred PredicateExpr) (renderResult, error) {
|
||||
// renderTagComprehension evaluates every tag predicate against individual JSON
|
||||
// string elements. This avoids JSON text matching and gives all comprehension
|
||||
// kinds the same exact, case-sensitive semantics.
|
||||
func (r *renderer) renderTagComprehension(field Field, pred PredicateExpr, kind ComprehensionKind) (renderResult, error) {
|
||||
arrayExpr := jsonArrayExpr(r.dialect, field)
|
||||
elemCond, err := r.elementPredicateSQL(pred)
|
||||
elemCond, err := r.tagElementPredicateSQL("tag_item.value", pred)
|
||||
if err != nil {
|
||||
return renderResult{}, err
|
||||
}
|
||||
|
||||
var elements, length string
|
||||
switch r.dialect {
|
||||
case DialectSQLite:
|
||||
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND %s != '[]'", arrayExpr, arrayExpr)
|
||||
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM json_each(%s) WHERE NOT (%s))", arrayExpr, elemCond)
|
||||
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
|
||||
arrayExpr = fmt.Sprintf("COALESCE(%s, JSON_ARRAY())", arrayExpr)
|
||||
elements = fmt.Sprintf("json_each(%s) AS tag_item", arrayExpr)
|
||||
length = fmt.Sprintf("json_array_length(%s)", arrayExpr)
|
||||
case DialectMySQL:
|
||||
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND JSON_LENGTH(%s) > 0", arrayExpr, arrayExpr)
|
||||
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM JSON_TABLE(%s, '$[*]' COLUMNS (value VARCHAR(512) PATH '$')) AS elem WHERE NOT (%s))", arrayExpr, elemCond)
|
||||
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
|
||||
arrayExpr = fmt.Sprintf("COALESCE(%s, JSON_ARRAY())", arrayExpr)
|
||||
elements = fmt.Sprintf("JSON_TABLE(%s, '$[*]' COLUMNS (value LONGTEXT PATH '$')) AS tag_item", arrayExpr)
|
||||
length = fmt.Sprintf("JSON_LENGTH(%s)", arrayExpr)
|
||||
case DialectPostgres:
|
||||
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND jsonb_array_length(%s) > 0", arrayExpr, arrayExpr)
|
||||
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(%s) AS elem(value) WHERE NOT (%s))", arrayExpr, elemCond)
|
||||
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
|
||||
arrayExpr = fmt.Sprintf("COALESCE(%s, '[]'::jsonb)", arrayExpr)
|
||||
elements = fmt.Sprintf("jsonb_array_elements_text(%s) AS tag_item(value)", arrayExpr)
|
||||
length = fmt.Sprintf("jsonb_array_length(%s)", arrayExpr)
|
||||
default:
|
||||
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
|
||||
}
|
||||
}
|
||||
|
||||
// renderTagExistsOne renders tags.exists_one(t, <pred>): exactly one element
|
||||
// satisfies the predicate, via a COUNT(...) = 1 subquery. A null or empty array
|
||||
// yields COUNT 0, which is correctly not equal to 1.
|
||||
func (r *renderer) renderTagExistsOne(field Field, pred PredicateExpr) (renderResult, error) {
|
||||
arrayExpr := jsonArrayExpr(r.dialect, field)
|
||||
elemCond, err := r.elementPredicateSQL(pred)
|
||||
if err != nil {
|
||||
return renderResult{}, err
|
||||
}
|
||||
switch r.dialect {
|
||||
case DialectSQLite:
|
||||
return renderResult{sql: fmt.Sprintf("(SELECT COUNT(*) FROM json_each(%s) WHERE %s) = 1", arrayExpr, elemCond)}, nil
|
||||
case DialectMySQL:
|
||||
return renderResult{sql: fmt.Sprintf("(SELECT COUNT(*) FROM JSON_TABLE(%s, '$[*]' COLUMNS (value VARCHAR(512) PATH '$')) AS elem WHERE %s) = 1", arrayExpr, elemCond)}, nil
|
||||
case DialectPostgres:
|
||||
return renderResult{sql: fmt.Sprintf("(SELECT COUNT(*) FROM jsonb_array_elements_text(%s) AS elem(value) WHERE %s) = 1", arrayExpr, elemCond)}, nil
|
||||
switch kind {
|
||||
case ComprehensionExists:
|
||||
return renderResult{sql: fmt.Sprintf("EXISTS (SELECT 1 FROM %s WHERE %s)", elements, elemCond)}, nil
|
||||
case ComprehensionAll:
|
||||
return renderResult{sql: fmt.Sprintf("(%s > 0 AND NOT EXISTS (SELECT 1 FROM %s WHERE NOT (%s)))", length, elements, elemCond)}, nil
|
||||
case ComprehensionExistsOne:
|
||||
return renderResult{sql: fmt.Sprintf("(SELECT COUNT(*) FROM %s WHERE %s) = 1", elements, elemCond)}, nil
|
||||
default:
|
||||
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
|
||||
return renderResult{}, errors.Errorf("unsupported comprehension kind %s", kind)
|
||||
}
|
||||
}
|
||||
|
||||
// elementPredicateSQL builds the per-element SQL condition for an all() predicate.
|
||||
// The iterated element is exposed as the unqualified column `value` on all dialects
|
||||
// (json_each.value / JSON_TABLE column / elem(value)).
|
||||
func (r *renderer) elementPredicateSQL(pred PredicateExpr) (string, error) {
|
||||
// tagElementPredicateSQL renders an exact, case-sensitive predicate for one tag.
|
||||
// SQLite's LIKE is case-insensitive for ASCII, so it uses string functions;
|
||||
// MySQL and Postgres use binary/C-collated LIKE with an explicit escape byte.
|
||||
func (r *renderer) tagElementPredicateSQL(element string, pred PredicateExpr) (string, error) {
|
||||
switch p := pred.(type) {
|
||||
case *EqualsPredicate:
|
||||
return fmt.Sprintf("value = %s", r.addArg(p.Value)), nil
|
||||
placeholder := r.addArg(p.Value)
|
||||
switch r.dialect {
|
||||
case DialectSQLite:
|
||||
return fmt.Sprintf("(%s COLLATE BINARY) = (%s COLLATE BINARY)", element, placeholder), nil
|
||||
case DialectMySQL:
|
||||
return fmt.Sprintf("CAST(%s AS BINARY) = CAST(%s AS BINARY)", element, placeholder), nil
|
||||
case DialectPostgres:
|
||||
return fmt.Sprintf("(%s COLLATE \"C\") = (%s::text COLLATE \"C\")", element, placeholder), nil
|
||||
}
|
||||
case *StartsWithPredicate:
|
||||
return r.foldedLike("value", likePattern(TextMatchPrefix, p.Prefix)), nil
|
||||
return r.tagElementTextMatch(element, TextMatchPrefix, p.Prefix)
|
||||
case *EndsWithPredicate:
|
||||
return r.foldedLike("value", likePattern(TextMatchSuffix, p.Suffix)), nil
|
||||
return r.tagElementTextMatch(element, TextMatchSuffix, p.Suffix)
|
||||
case *ContainsPredicate:
|
||||
return r.foldedLike("value", likePattern(TextMatchContains, p.Substring)), nil
|
||||
return r.tagElementTextMatch(element, TextMatchContains, p.Substring)
|
||||
default:
|
||||
return "", errors.Errorf("unsupported predicate %T in all()", pred)
|
||||
return "", errors.Errorf("unsupported tag predicate %T", pred)
|
||||
}
|
||||
return "", errors.Errorf("unsupported dialect %s", r.dialect)
|
||||
}
|
||||
|
||||
// renderTagEquals generates SQL for tags.exists(t, t == "value").
|
||||
func (r *renderer) renderTagEquals(field Field, value string, _ ComprehensionKind) (renderResult, error) {
|
||||
arrayExpr := jsonArrayExpr(r.dialect, field)
|
||||
|
||||
switch r.dialect {
|
||||
case DialectSQLite, DialectMySQL:
|
||||
exactMatch := r.buildJSONArrayLike(arrayExpr, fmt.Sprintf(`%%"%s"%%`, value))
|
||||
return renderResult{sql: r.wrapWithNullCheck(arrayExpr, exactMatch)}, nil
|
||||
case DialectPostgres:
|
||||
exactMatch := fmt.Sprintf("%s @> jsonb_build_array(%s::json)", arrayExpr, r.addArg(fmt.Sprintf(`"%s"`, value)))
|
||||
return renderResult{sql: r.wrapWithNullCheck(arrayExpr, exactMatch)}, nil
|
||||
default:
|
||||
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
|
||||
func (r *renderer) tagElementTextMatch(element string, mode TextMatchMode, value string) (string, error) {
|
||||
if value == "" {
|
||||
return "1 = 1", nil
|
||||
}
|
||||
}
|
||||
|
||||
// renderTagStartsWith generates SQL for tags.exists(t, t.startsWith("prefix")).
|
||||
func (r *renderer) renderTagStartsWith(field Field, prefix string, _ ComprehensionKind) (renderResult, error) {
|
||||
arrayExpr := jsonArrayExpr(r.dialect, field)
|
||||
|
||||
switch r.dialect {
|
||||
case DialectSQLite, DialectMySQL:
|
||||
// Match exact tag or tags with this prefix (hierarchical support)
|
||||
exactMatch := r.buildJSONArrayLike(arrayExpr, fmt.Sprintf(`%%"%s"%%`, prefix))
|
||||
prefixMatch := r.buildJSONArrayLike(arrayExpr, fmt.Sprintf(`%%"%s%%`, prefix))
|
||||
condition := fmt.Sprintf("(%s OR %s)", exactMatch, prefixMatch)
|
||||
return renderResult{sql: r.wrapWithNullCheck(arrayExpr, condition)}, nil
|
||||
|
||||
case DialectPostgres:
|
||||
// Use PostgreSQL's powerful JSON operators
|
||||
exactMatch := fmt.Sprintf("%s @> jsonb_build_array(%s::json)", arrayExpr, r.addArg(fmt.Sprintf(`"%s"`, prefix)))
|
||||
prefixMatch := fmt.Sprintf("(%s)::text LIKE %s", arrayExpr, r.addArg(fmt.Sprintf(`%%"%s%%`, prefix)))
|
||||
condition := fmt.Sprintf("(%s OR %s)", exactMatch, prefixMatch)
|
||||
return renderResult{sql: r.wrapWithNullCheck(arrayExpr, condition)}, nil
|
||||
|
||||
default:
|
||||
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
|
||||
}
|
||||
}
|
||||
|
||||
// renderTagEndsWith generates SQL for tags.exists(t, t.endsWith("suffix")).
|
||||
func (r *renderer) renderTagEndsWith(field Field, suffix string, _ ComprehensionKind) (renderResult, error) {
|
||||
arrayExpr := jsonArrayExpr(r.dialect, field)
|
||||
pattern := fmt.Sprintf(`%%%s"%%`, suffix)
|
||||
|
||||
likeExpr := r.buildJSONArrayLike(arrayExpr, pattern)
|
||||
return renderResult{sql: r.wrapWithNullCheck(arrayExpr, likeExpr)}, nil
|
||||
}
|
||||
|
||||
// renderTagContains generates SQL for tags.exists(t, t.contains("substring")).
|
||||
func (r *renderer) renderTagContains(field Field, substring string, _ ComprehensionKind) (renderResult, error) {
|
||||
arrayExpr := jsonArrayExpr(r.dialect, field)
|
||||
pattern := fmt.Sprintf(`%%%s%%`, substring)
|
||||
|
||||
likeExpr := r.buildJSONArrayLike(arrayExpr, pattern)
|
||||
return renderResult{sql: r.wrapWithNullCheck(arrayExpr, likeExpr)}, nil
|
||||
}
|
||||
|
||||
// buildJSONArrayLike builds a LIKE expression for matching within a JSON array.
|
||||
// Returns the LIKE clause without NULL/empty checks.
|
||||
func (r *renderer) buildJSONArrayLike(arrayExpr, pattern string) string {
|
||||
switch r.dialect {
|
||||
case DialectSQLite, DialectMySQL:
|
||||
return fmt.Sprintf("%s LIKE %s", arrayExpr, r.addArg(pattern))
|
||||
case DialectPostgres:
|
||||
return fmt.Sprintf("(%s)::text LIKE %s", arrayExpr, r.addArg(pattern))
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// wrapWithNullCheck wraps a condition with NULL and empty array checks.
|
||||
// This ensures we don't match against NULL or empty JSON arrays.
|
||||
func (r *renderer) wrapWithNullCheck(arrayExpr, condition string) string {
|
||||
var nullCheck string
|
||||
switch r.dialect {
|
||||
case DialectSQLite:
|
||||
nullCheck = fmt.Sprintf("%s IS NOT NULL AND %s != '[]'", arrayExpr, arrayExpr)
|
||||
switch mode {
|
||||
case TextMatchPrefix:
|
||||
return fmt.Sprintf("instr(%s, %s) = 1", element, r.addArg(value)), nil
|
||||
case TextMatchSuffix:
|
||||
lengthValue := r.addArg(value)
|
||||
compareValue := r.addArg(value)
|
||||
return fmt.Sprintf("(substr(%s, -length(%s)) COLLATE BINARY) = (%s COLLATE BINARY)", element, lengthValue, compareValue), nil
|
||||
case TextMatchContains:
|
||||
return fmt.Sprintf("instr(%s, %s) > 0", element, r.addArg(value)), nil
|
||||
default:
|
||||
return "", errors.Errorf("unsupported tag text match mode %s", mode)
|
||||
}
|
||||
case DialectMySQL:
|
||||
nullCheck = fmt.Sprintf("%s IS NOT NULL AND JSON_LENGTH(%s) > 0", arrayExpr, arrayExpr)
|
||||
pattern := tagLikePattern(mode, value)
|
||||
return fmt.Sprintf("CAST(%s AS BINARY) LIKE CAST(%s AS BINARY) ESCAPE '!'", element, r.addArg(pattern)), nil
|
||||
case DialectPostgres:
|
||||
nullCheck = fmt.Sprintf("%s IS NOT NULL AND jsonb_array_length(%s) > 0", arrayExpr, arrayExpr)
|
||||
pattern := tagLikePattern(mode, value)
|
||||
return fmt.Sprintf("(%s COLLATE \"C\") LIKE (%s::text COLLATE \"C\") ESCAPE '!'", element, r.addArg(pattern)), nil
|
||||
default:
|
||||
return condition
|
||||
return "", errors.Errorf("unsupported dialect %s", r.dialect)
|
||||
}
|
||||
}
|
||||
|
||||
func tagLikePattern(mode TextMatchMode, value string) string {
|
||||
escaped := strings.NewReplacer("!", "!!", "%", "!%", "_", "!_").Replace(value)
|
||||
switch mode {
|
||||
case TextMatchPrefix:
|
||||
return escaped + "%"
|
||||
case TextMatchSuffix:
|
||||
return "%" + escaped
|
||||
default:
|
||||
return "%" + escaped + "%"
|
||||
}
|
||||
return fmt.Sprintf("(%s AND %s)", condition, nullCheck)
|
||||
}
|
||||
|
||||
func (r *renderer) jsonBoolPredicate(field Field) (string, error) {
|
||||
|
|
|
|||
29
internal/markdown/ast/gfm_email.go
Normal file
29
internal/markdown/ast/gfm_email.go
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
package ast
|
||||
|
||||
import gast "github.com/yuin/goldmark/ast"
|
||||
|
||||
// GFMEmailNode represents a written extended GFM email autolink.
|
||||
type GFMEmailNode struct {
|
||||
gast.BaseInline
|
||||
|
||||
// Source is the exact Markdown source spelling.
|
||||
Source []byte
|
||||
// Address is the decoded email address used by the rendered link.
|
||||
Address []byte
|
||||
}
|
||||
|
||||
// KindGFMEmail is the NodeKind for GFMEmailNode.
|
||||
var KindGFMEmail = gast.NewNodeKind("GFMEmail")
|
||||
|
||||
// Kind returns KindGFMEmail.
|
||||
func (*GFMEmailNode) Kind() gast.NodeKind {
|
||||
return KindGFMEmail
|
||||
}
|
||||
|
||||
// Dump implements Node.Dump for debugging.
|
||||
func (n *GFMEmailNode) Dump(source []byte, level int) {
|
||||
gast.DumpHelper(n, source, level, map[string]string{
|
||||
"Address": string(n.Address),
|
||||
"Source": string(n.Source),
|
||||
}, nil)
|
||||
}
|
||||
78
internal/markdown/ast/math.go
Normal file
78
internal/markdown/ast/math.go
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
package ast
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
gast "github.com/yuin/goldmark/ast"
|
||||
)
|
||||
|
||||
// InlineMathNode represents literal inline math, including its dollar delimiters.
|
||||
type InlineMathNode struct {
|
||||
gast.BaseInline
|
||||
|
||||
// Source is the complete logical source spelling.
|
||||
Source []byte
|
||||
}
|
||||
|
||||
// KindInlineMath is the NodeKind for InlineMathNode.
|
||||
var KindInlineMath = gast.NewNodeKind("InlineMath")
|
||||
|
||||
// Kind returns KindInlineMath.
|
||||
func (*InlineMathNode) Kind() gast.NodeKind {
|
||||
return KindInlineMath
|
||||
}
|
||||
|
||||
// Dump implements Node.Dump for debugging.
|
||||
func (n *InlineMathNode) Dump(source []byte, level int) {
|
||||
gast.DumpHelper(n, source, level, map[string]string{
|
||||
"Source": string(n.Source),
|
||||
}, nil)
|
||||
}
|
||||
|
||||
// BlockMathNode represents literal flow math, including its dollar fences.
|
||||
type BlockMathNode struct {
|
||||
gast.BaseBlock
|
||||
|
||||
// Source is the complete logical source spelling.
|
||||
Source []byte
|
||||
fence int
|
||||
}
|
||||
|
||||
// KindBlockMath is the NodeKind for BlockMathNode.
|
||||
var KindBlockMath = gast.NewNodeKind("BlockMath")
|
||||
|
||||
// NewBlockMathNode creates a flow math node from its opening line.
|
||||
func NewBlockMathNode(opening []byte, fence int) *BlockMathNode {
|
||||
return &BlockMathNode{
|
||||
Source: append([]byte(nil), opening...),
|
||||
fence: fence,
|
||||
}
|
||||
}
|
||||
|
||||
// AppendSource appends one logical source line to the node.
|
||||
func (n *BlockMathNode) AppendSource(line []byte) {
|
||||
n.Source = append(n.Source, line...)
|
||||
}
|
||||
|
||||
// FenceLength returns the opening dollar fence length.
|
||||
func (n *BlockMathNode) FenceLength() int {
|
||||
return n.fence
|
||||
}
|
||||
|
||||
// IsRaw prevents Goldmark from parsing math contents as Markdown.
|
||||
func (*BlockMathNode) IsRaw() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// Kind returns KindBlockMath.
|
||||
func (*BlockMathNode) Kind() gast.NodeKind {
|
||||
return KindBlockMath
|
||||
}
|
||||
|
||||
// Dump implements Node.Dump for debugging.
|
||||
func (n *BlockMathNode) Dump(source []byte, level int) {
|
||||
gast.DumpHelper(n, source, level, map[string]string{
|
||||
"FenceLength": fmt.Sprintf("%d", n.fence),
|
||||
"Source": string(n.Source),
|
||||
}, nil)
|
||||
}
|
||||
|
|
@ -8,8 +8,10 @@ import (
|
|||
type TagNode struct {
|
||||
gast.BaseInline
|
||||
|
||||
// Tag name without the # prefix
|
||||
// Tag is the emitted tag value without the # prefix.
|
||||
Tag []byte
|
||||
// Source is the complete recognized source span, including the # prefix.
|
||||
Source []byte
|
||||
}
|
||||
|
||||
// KindTag is the NodeKind for TagNode.
|
||||
|
|
@ -23,6 +25,7 @@ func (*TagNode) Kind() gast.NodeKind {
|
|||
// Dump implements Node.Dump for debugging.
|
||||
func (n *TagNode) Dump(source []byte, level int) {
|
||||
gast.DumpHelper(n, source, level, map[string]string{
|
||||
"Tag": string(n.Tag),
|
||||
"Tag": string(n.Tag),
|
||||
"Source": string(n.Source),
|
||||
}, nil)
|
||||
}
|
||||
|
|
|
|||
323
internal/markdown/extensions/gfm.go
Normal file
323
internal/markdown/extensions/gfm.go
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
package extensions
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"regexp"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/yuin/goldmark"
|
||||
"github.com/yuin/goldmark/ast"
|
||||
goldmarkextension "github.com/yuin/goldmark/extension"
|
||||
"github.com/yuin/goldmark/parser"
|
||||
"github.com/yuin/goldmark/renderer"
|
||||
"github.com/yuin/goldmark/text"
|
||||
"github.com/yuin/goldmark/util"
|
||||
|
||||
mast "github.com/usememos/memos/internal/markdown/ast"
|
||||
mparser "github.com/usememos/memos/internal/markdown/parser"
|
||||
)
|
||||
|
||||
const gfmDomainPattern = `(?:[\p{L}\p{N}_-]+\.)*[\p{L}\p{N}-]+\.[\p{L}\p{N}-]+`
|
||||
|
||||
var (
|
||||
gfmURLPattern = regexp.MustCompile(`^https?://` + gfmDomainPattern + `[^<\x09-\x0d\p{Z}]*`)
|
||||
gfmWWWPattern = regexp.MustCompile(`^www\.` + gfmDomainPattern + `[^<\x09-\x0d\p{Z}]*`)
|
||||
// Extended emails are resolved after emphasis instead of by the linkify parser.
|
||||
gfmNoEmail = regexp.MustCompile(`a^`)
|
||||
)
|
||||
|
||||
type gfmLinkify struct{}
|
||||
|
||||
// NewGFMLinkify returns a linkifier configured for written GFM 0.29.
|
||||
func NewGFMLinkify() goldmark.Extender {
|
||||
return &gfmLinkify{}
|
||||
}
|
||||
|
||||
func (*gfmLinkify) Extend(markdown goldmark.Markdown) {
|
||||
markdown.Parser().AddOptions(
|
||||
parser.WithInlineParsers(util.Prioritized(newGFMLinkifyParser(), 999)),
|
||||
parser.WithASTTransformers(
|
||||
util.Prioritized(&gfmLinkifyASTTransformer{}, 900),
|
||||
util.Prioritized(&gfmEmailASTTransformer{}, 950),
|
||||
),
|
||||
)
|
||||
markdown.Renderer().AddOptions(
|
||||
renderer.WithNodeRenderers(
|
||||
util.Prioritized(&gfmEmailNodeRenderer{}, 500),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
type gfmEmailNodeRenderer struct{}
|
||||
|
||||
func (*gfmEmailNodeRenderer) RegisterFuncs(registerer renderer.NodeRendererFuncRegisterer) {
|
||||
registerer.Register(mast.KindGFMEmail, renderGFMEmailNode)
|
||||
}
|
||||
|
||||
func renderGFMEmailNode(writer util.BufWriter, _ []byte, node ast.Node, entering bool) (ast.WalkStatus, error) {
|
||||
if !entering {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
emailNode, ok := node.(*mast.GFMEmailNode)
|
||||
if !ok {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
_, _ = writer.WriteString(`<a href="mailto:`)
|
||||
_, _ = writer.Write(util.EscapeHTML(emailNode.Address))
|
||||
_, _ = writer.WriteString(`">`)
|
||||
_, _ = writer.Write(util.EscapeHTML(emailNode.Address))
|
||||
_, _ = writer.WriteString(`</a>`)
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
|
||||
func newGFMLinkifyParser() parser.InlineParser {
|
||||
return goldmarkextension.NewLinkifyParser(
|
||||
goldmarkextension.WithLinkifyAllowedProtocols([]string{"http:", "https:"}),
|
||||
goldmarkextension.WithLinkifyURLRegexp(gfmURLPattern),
|
||||
goldmarkextension.WithLinkifyWWWRegexp(gfmWWWPattern),
|
||||
// Extended emails are resolved after emphasis delimiters so unmatched
|
||||
// underscores are consolidated into their surrounding text node first.
|
||||
goldmarkextension.WithLinkifyEmailRegexp(gfmNoEmail),
|
||||
)
|
||||
}
|
||||
|
||||
type gfmLinkifyASTTransformer struct{}
|
||||
|
||||
func (*gfmLinkifyASTTransformer) Transform(document *ast.Document, reader text.Reader, _ parser.Context) {
|
||||
var invalid []*ast.AutoLink
|
||||
_ = ast.Walk(document, func(node ast.Node, entering bool) (ast.WalkStatus, error) {
|
||||
link, ok := node.(*ast.AutoLink)
|
||||
if !entering || !ok || link.AutoLinkType != ast.AutoLinkURL {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
label := link.Label(reader.Source())
|
||||
start, found := autoLinkLabelStart(link, reader.Source(), label)
|
||||
if !found || isAngleAutoLink(reader.Source(), start, len(label)) {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
if !hasValidGFMDomain(label) {
|
||||
invalid = append(invalid, link)
|
||||
}
|
||||
return ast.WalkContinue, nil
|
||||
})
|
||||
|
||||
for _, link := range invalid {
|
||||
parent := link.Parent()
|
||||
if parent == nil {
|
||||
continue
|
||||
}
|
||||
label := link.Label(reader.Source())
|
||||
start, found := autoLinkLabelStart(link, reader.Source(), label)
|
||||
if !found {
|
||||
continue
|
||||
}
|
||||
parent.ReplaceChild(parent, link, ast.NewTextSegment(text.NewSegment(start, start+len(label))))
|
||||
}
|
||||
|
||||
replaceInitialBOMURL(document, reader.Source())
|
||||
replaceUnparsedGFMURLs(document, reader.Source())
|
||||
}
|
||||
|
||||
type inlineNodeMatch struct {
|
||||
start int
|
||||
end int
|
||||
node ast.Node
|
||||
}
|
||||
|
||||
func eligibleLiteralTextNodes(document *ast.Document) []*ast.Text {
|
||||
var nodes []*ast.Text
|
||||
_ = ast.Walk(document, func(node ast.Node, entering bool) (ast.WalkStatus, error) {
|
||||
textNode, ok := node.(*ast.Text)
|
||||
if entering && ok && isEligibleLiteralText(textNode) {
|
||||
nodes = append(nodes, textNode)
|
||||
}
|
||||
return ast.WalkContinue, nil
|
||||
})
|
||||
return nodes
|
||||
}
|
||||
|
||||
func replaceInitialBOMURL(document *ast.Document, source []byte) {
|
||||
const bomLength = len("\uFEFF")
|
||||
if !bytes.HasPrefix(source, []byte("\uFEFF")) {
|
||||
return
|
||||
}
|
||||
|
||||
linkParser := newGFMLinkifyParser()
|
||||
for _, textNode := range eligibleLiteralTextNodes(document) {
|
||||
segment := textNode.Segment
|
||||
if segment.Start > bomLength || segment.Stop <= bomLength || textNode.Parent() == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
urlReader := text.NewReader(source[:segment.Stop])
|
||||
urlReader.Advance(bomLength)
|
||||
link, ok := linkParser.Parse(textNode.Parent(), urlReader, parser.NewContext()).(*ast.AutoLink)
|
||||
if !ok || link.AutoLinkType != ast.AutoLinkURL {
|
||||
return
|
||||
}
|
||||
label := link.Label(source)
|
||||
if !hasValidGFMDomain(label) {
|
||||
return
|
||||
}
|
||||
end := bomLength + len(label)
|
||||
link.SetPos(bomLength)
|
||||
replaceInlineMatches(textNode, []inlineNodeMatch{{start: bomLength, end: end, node: link}})
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
func replaceUnparsedGFMURLs(document *ast.Document, source []byte) {
|
||||
linkParser := newGFMLinkifyParser()
|
||||
textNodes := eligibleLiteralTextNodes(document)
|
||||
mergeAdjacentLiteralText(textNodes, source)
|
||||
for _, textNode := range textNodes {
|
||||
if textNode.Parent() == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
segment := textNode.Segment
|
||||
var matches []inlineNodeMatch
|
||||
for offset := segment.Start; offset < segment.Stop; offset++ {
|
||||
if !hasWrittenGFMURLBoundary(source, offset) {
|
||||
continue
|
||||
}
|
||||
|
||||
urlReader := text.NewReader(source[:segment.Stop])
|
||||
urlReader.Advance(offset)
|
||||
link, ok := linkParser.Parse(textNode.Parent(), urlReader, parser.NewContext()).(*ast.AutoLink)
|
||||
if !ok || link.AutoLinkType != ast.AutoLinkURL {
|
||||
continue
|
||||
}
|
||||
label := link.Label(source)
|
||||
if len(label) == 0 || offset+len(label) > segment.Stop ||
|
||||
!bytes.HasPrefix(source[offset:segment.Stop], label) || !hasValidGFMDomain(label) {
|
||||
continue
|
||||
}
|
||||
|
||||
start := offset
|
||||
end := start + len(label)
|
||||
link.SetPos(start)
|
||||
matches = append(matches, inlineNodeMatch{start: start, end: end, node: link})
|
||||
offset = end - 1
|
||||
}
|
||||
replaceInlineMatches(textNode, matches)
|
||||
}
|
||||
}
|
||||
|
||||
func hasWrittenGFMURLBoundary(source []byte, offset int) bool {
|
||||
if offset == 0 {
|
||||
return true
|
||||
}
|
||||
previous := source[offset-1]
|
||||
return previous == ' ' || previous >= '\t' && previous <= '\r' ||
|
||||
previous == '*' || previous == '_' || previous == '~' || previous == '('
|
||||
}
|
||||
|
||||
func autoLinkLabelStart(link *ast.AutoLink, source, label []byte) (int, bool) {
|
||||
start := link.Pos()
|
||||
if start >= 0 && start <= len(source)-len(label) && bytes.HasPrefix(source[start:], label) {
|
||||
return start, true
|
||||
}
|
||||
// Goldmark gives a written link triggered by preceding punctuation that
|
||||
// punctuation's position, although it is not part of the label.
|
||||
start++
|
||||
if start >= 0 && start <= len(source)-len(label) && bytes.HasPrefix(source[start:], label) {
|
||||
return start, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
func isAngleAutoLink(source []byte, labelStart, labelLength int) bool {
|
||||
return labelStart > 0 && labelStart+labelLength < len(source) &&
|
||||
source[labelStart-1] == '<' && source[labelStart+labelLength] == '>'
|
||||
}
|
||||
|
||||
func hasValidGFMDomain(label []byte) bool {
|
||||
domainStart := 0
|
||||
switch {
|
||||
case bytes.HasPrefix(label, []byte("http://")):
|
||||
domainStart = len("http://")
|
||||
case bytes.HasPrefix(label, []byte("https://")):
|
||||
domainStart = len("https://")
|
||||
case bytes.HasPrefix(label, []byte("www.")):
|
||||
domainStart = len("www.")
|
||||
default:
|
||||
return false
|
||||
}
|
||||
|
||||
domainEnd := domainStart
|
||||
for domainEnd < len(label) {
|
||||
r, size := utf8.DecodeRune(label[domainEnd:])
|
||||
if (r == utf8.RuneError && size == 1) || !isGFMDomainRune(r) {
|
||||
break
|
||||
}
|
||||
domainEnd += size
|
||||
}
|
||||
segments := bytes.Split(label[domainStart:domainEnd], []byte{'.'})
|
||||
if len(segments) < 2 {
|
||||
return false
|
||||
}
|
||||
return !bytes.ContainsRune(segments[len(segments)-2], '_') && !bytes.ContainsRune(segments[len(segments)-1], '_')
|
||||
}
|
||||
|
||||
func isGFMDomainRune(value rune) bool {
|
||||
return unicode.IsLetter(value) || unicode.IsNumber(value) || value == '_' || value == '-' || value == '.'
|
||||
}
|
||||
|
||||
type gfmEmailASTTransformer struct{}
|
||||
|
||||
func (*gfmEmailASTTransformer) Transform(document *ast.Document, reader text.Reader, _ parser.Context) {
|
||||
source := reader.Source()
|
||||
textNodes := eligibleLiteralTextNodes(document)
|
||||
mergeAdjacentLiteralText(textNodes, source)
|
||||
for _, textNode := range textNodes {
|
||||
if textNode.Parent() != nil {
|
||||
replaceGFMEmailsInText(textNode, source)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func replaceGFMEmailsInText(textNode *ast.Text, source []byte) {
|
||||
segment := textNode.Segment
|
||||
emailMatches := mparser.FindGFMEmailMatches(source[segment.Start:segment.Stop])
|
||||
matches := make([]inlineNodeMatch, 0, len(emailMatches))
|
||||
for _, match := range emailMatches {
|
||||
start := segment.Start + match.Start
|
||||
end := segment.Start + match.End
|
||||
emailNode := &mast.GFMEmailNode{
|
||||
Source: append([]byte(nil), source[start:end]...),
|
||||
Address: append([]byte(nil), match.Address...),
|
||||
}
|
||||
emailNode.SetPos(start)
|
||||
matches = append(matches, inlineNodeMatch{
|
||||
start: start,
|
||||
end: end,
|
||||
node: emailNode,
|
||||
})
|
||||
}
|
||||
replaceInlineMatches(textNode, matches)
|
||||
}
|
||||
|
||||
func replaceInlineMatches(textNode *ast.Text, matches []inlineNodeMatch) {
|
||||
if len(matches) == 0 {
|
||||
return
|
||||
}
|
||||
segment := textNode.Segment
|
||||
parent := textNode.Parent()
|
||||
cursor := segment.Start
|
||||
padding := segment.Padding
|
||||
for _, match := range matches {
|
||||
if match.start > cursor || padding > 0 {
|
||||
insertSplitTextBefore(parent, textNode, textNode, text.NewSegmentPadding(cursor, match.start, padding), false)
|
||||
}
|
||||
parent.InsertBefore(parent, textNode, match.node)
|
||||
cursor = match.end
|
||||
padding = 0
|
||||
}
|
||||
|
||||
if cursor < segment.Stop || textNode.SoftLineBreak() || textNode.HardLineBreak() {
|
||||
insertSplitTextBefore(parent, textNode, textNode, text.NewSegment(cursor, segment.Stop), true)
|
||||
}
|
||||
parent.RemoveChild(parent, textNode)
|
||||
}
|
||||
|
|
@ -2,9 +2,14 @@ package extensions
|
|||
|
||||
import (
|
||||
"github.com/yuin/goldmark"
|
||||
"github.com/yuin/goldmark/ast"
|
||||
east "github.com/yuin/goldmark/extension/ast"
|
||||
"github.com/yuin/goldmark/parser"
|
||||
"github.com/yuin/goldmark/renderer"
|
||||
"github.com/yuin/goldmark/text"
|
||||
"github.com/yuin/goldmark/util"
|
||||
|
||||
mast "github.com/usememos/memos/internal/markdown/ast"
|
||||
mparser "github.com/usememos/memos/internal/markdown/parser"
|
||||
)
|
||||
|
||||
|
|
@ -16,9 +21,161 @@ var TagExtension = &tagExtension{}
|
|||
// Extend extends the goldmark parser with tag support.
|
||||
func (*tagExtension) Extend(m goldmark.Markdown) {
|
||||
m.Parser().AddOptions(
|
||||
parser.WithBlockParsers(
|
||||
util.Prioritized(mparser.NewBlockMathParser(), 750),
|
||||
),
|
||||
parser.WithInlineParsers(
|
||||
// Priority 200 - run before standard link parser (500)
|
||||
util.Prioritized(mparser.NewTagParser(), 200),
|
||||
util.Prioritized(mparser.NewInlineMathParser(), 150),
|
||||
),
|
||||
parser.WithASTTransformers(
|
||||
// Run after GFM has resolved emphasis, links, tables, and other syntax.
|
||||
util.Prioritized(&tagASTTransformer{}, 1000),
|
||||
),
|
||||
)
|
||||
m.Renderer().AddOptions(
|
||||
renderer.WithNodeRenderers(
|
||||
util.Prioritized(&tagNodeRenderer{}, 500),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
type tagASTTransformer struct{}
|
||||
|
||||
func (*tagASTTransformer) Transform(document *ast.Document, reader text.Reader, _ parser.Context) {
|
||||
source := reader.Source()
|
||||
textNodes := eligibleLiteralTextNodes(document)
|
||||
mergeAdjacentLiteralText(textNodes, source)
|
||||
for _, textNode := range textNodes {
|
||||
if textNode.Parent() == nil {
|
||||
continue
|
||||
}
|
||||
replaceTagsInText(textNode, source)
|
||||
}
|
||||
}
|
||||
|
||||
func mergeAdjacentLiteralText(textNodes []*ast.Text, source []byte) {
|
||||
for _, textNode := range textNodes {
|
||||
parent := textNode.Parent()
|
||||
if parent == nil {
|
||||
continue
|
||||
}
|
||||
for {
|
||||
nextText, ok := textNode.NextSibling().(*ast.Text)
|
||||
if !ok || !isEligibleLiteralText(nextText) || !textNode.Merge(nextText, source) {
|
||||
break
|
||||
}
|
||||
parent.RemoveChild(parent, nextText)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func isEligibleLiteralText(textNode *ast.Text) bool {
|
||||
if textNode.IsRaw() {
|
||||
return false
|
||||
}
|
||||
for parent := textNode.Parent(); parent != nil; parent = parent.Parent() {
|
||||
switch parent.Kind() {
|
||||
case ast.KindDocument,
|
||||
ast.KindTextBlock,
|
||||
ast.KindParagraph,
|
||||
ast.KindHeading,
|
||||
ast.KindEmphasis,
|
||||
ast.KindBlockquote,
|
||||
ast.KindList,
|
||||
ast.KindListItem,
|
||||
east.KindStrikethrough,
|
||||
east.KindTable,
|
||||
east.KindTableRow,
|
||||
east.KindTableHeader,
|
||||
east.KindTableCell:
|
||||
// Explicitly transparent GFM text containers.
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func replaceTagsInText(textNode *ast.Text, source []byte) {
|
||||
segment := textNode.Segment
|
||||
matches := mparser.FindTagMatches(segment.Value(source))
|
||||
if len(matches) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
parent := textNode.Parent()
|
||||
cursor := segment.Start
|
||||
padding := segment.Padding
|
||||
for _, match := range matches {
|
||||
start := segment.Start + match.Start - segment.Padding
|
||||
end := segment.Start + match.End - segment.Padding
|
||||
if start > cursor || padding > 0 {
|
||||
insertSplitTextBefore(parent, textNode, textNode, text.NewSegmentPadding(cursor, start, padding), false)
|
||||
}
|
||||
|
||||
tagNode := &mast.TagNode{
|
||||
Tag: append([]byte(nil), match.Value...),
|
||||
Source: append([]byte(nil), source[start:end]...),
|
||||
}
|
||||
tagNode.SetPos(start)
|
||||
parent.InsertBefore(parent, textNode, tagNode)
|
||||
cursor = end
|
||||
padding = 0
|
||||
}
|
||||
|
||||
if cursor < segment.Stop || textNode.SoftLineBreak() || textNode.HardLineBreak() {
|
||||
insertSplitTextBefore(parent, textNode, textNode, text.NewSegment(cursor, segment.Stop), true)
|
||||
}
|
||||
parent.RemoveChild(parent, textNode)
|
||||
}
|
||||
|
||||
func insertSplitTextBefore(parent ast.Node, before ast.Node, original *ast.Text, segment text.Segment, keepLineBreak bool) {
|
||||
newText := ast.NewTextSegment(segment)
|
||||
newText.SetRaw(original.IsRaw())
|
||||
if keepLineBreak {
|
||||
newText.SetSoftLineBreak(original.SoftLineBreak())
|
||||
newText.SetHardLineBreak(original.HardLineBreak())
|
||||
}
|
||||
parent.InsertBefore(parent, before, newText)
|
||||
}
|
||||
|
||||
type tagNodeRenderer struct{}
|
||||
|
||||
func (*tagNodeRenderer) RegisterFuncs(registerer renderer.NodeRendererFuncRegisterer) {
|
||||
registerer.Register(mast.KindTag, renderTagNode)
|
||||
registerer.Register(mast.KindInlineMath, renderMathNode)
|
||||
registerer.Register(mast.KindBlockMath, renderMathNode)
|
||||
}
|
||||
|
||||
func renderTagNode(writer util.BufWriter, _ []byte, node ast.Node, entering bool) (ast.WalkStatus, error) {
|
||||
if !entering {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
tagNode, ok := node.(*mast.TagNode)
|
||||
if !ok {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
spelling := tagNode.Source
|
||||
if len(spelling) == 0 {
|
||||
spelling = append([]byte{'#'}, tagNode.Tag...)
|
||||
}
|
||||
_, _ = writer.Write(util.EscapeHTML(spelling))
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
|
||||
func renderMathNode(writer util.BufWriter, _ []byte, node ast.Node, entering bool) (ast.WalkStatus, error) {
|
||||
if !entering {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
var source []byte
|
||||
switch mathNode := node.(type) {
|
||||
case *mast.InlineMathNode:
|
||||
source = mathNode.Source
|
||||
case *mast.BlockMathNode:
|
||||
source = mathNode.Source
|
||||
default:
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
_, _ = writer.Write(util.EscapeHTML(source))
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
|
|
|
|||
76
internal/markdown/link_reference.go
Normal file
76
internal/markdown/link_reference.go
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
package markdown
|
||||
|
||||
import (
|
||||
"github.com/yuin/goldmark/ast"
|
||||
"github.com/yuin/goldmark/text"
|
||||
"github.com/yuin/goldmark/util"
|
||||
)
|
||||
|
||||
// maskInvalidLinkReferenceDefinitions hides reference definitions that
|
||||
// Goldmark accepts with an unclosed parenthesis in a bare destination.
|
||||
// CommonMark treats those lines as ordinary Markdown text.
|
||||
func maskInvalidLinkReferenceDefinitions(root ast.Node, source []byte) []byte {
|
||||
var masked []byte
|
||||
_ = ast.Walk(root, func(node ast.Node, entering bool) (ast.WalkStatus, error) {
|
||||
definition, ok := node.(*ast.LinkReferenceDefinition)
|
||||
if !entering || !ok || !hasUnclosedParenthesis(definition.Destination) {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
|
||||
colon, bare := linkReferenceDestination(source, definition)
|
||||
if !bare {
|
||||
return ast.WalkContinue, nil
|
||||
}
|
||||
if masked == nil {
|
||||
masked = append([]byte(nil), source...)
|
||||
}
|
||||
masked[colon] = ';'
|
||||
return ast.WalkContinue, nil
|
||||
})
|
||||
return masked
|
||||
}
|
||||
|
||||
func hasUnclosedParenthesis(destination []byte) bool {
|
||||
depth := 0
|
||||
for i := 0; i < len(destination); i++ {
|
||||
switch destination[i] {
|
||||
case '\\':
|
||||
if i+1 < len(destination) && util.IsPunct(destination[i+1]) {
|
||||
i++
|
||||
}
|
||||
case '(':
|
||||
depth++
|
||||
case ')':
|
||||
depth--
|
||||
default:
|
||||
// Other destination bytes do not affect parenthesis balance.
|
||||
}
|
||||
}
|
||||
return depth > 0
|
||||
}
|
||||
|
||||
// linkReferenceDestination returns the definition colon and whether its
|
||||
// destination uses the bare form whose parentheses must balance.
|
||||
func linkReferenceDestination(source []byte, definition *ast.LinkReferenceDefinition) (int, bool) {
|
||||
reader := text.NewBlockReader(source, definition.Lines())
|
||||
reader.SkipSpaces()
|
||||
if reader.Peek() != '[' {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
reader.Advance(1)
|
||||
closureOptions := text.FindClosureOptions{Nesting: false, Newline: true, Advance: true}
|
||||
if _, found := reader.FindClosure('[', ']', closureOptions); !found || reader.Peek() != ':' {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
_, position := reader.Position()
|
||||
colon := position.Start
|
||||
if colon < 0 || colon >= len(source) || source[colon] != ':' {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
reader.Advance(1)
|
||||
reader.SkipSpaces()
|
||||
return colon, reader.Peek() != '<'
|
||||
}
|
||||
|
|
@ -2,6 +2,8 @@ package markdown
|
|||
|
||||
import (
|
||||
"bytes"
|
||||
"cmp"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/yuin/goldmark"
|
||||
|
|
@ -89,7 +91,10 @@ func NewService(opts ...Option) Service {
|
|||
}
|
||||
|
||||
exts := []goldmark.Extender{
|
||||
extension.GFM, // GitHub Flavored Markdown (tables, strikethrough, task lists, autolinks)
|
||||
extension.Table,
|
||||
extension.Strikethrough,
|
||||
extension.TaskList,
|
||||
extensions.NewGFMLinkify(),
|
||||
}
|
||||
|
||||
// Add custom extensions based on config
|
||||
|
|
@ -116,6 +121,9 @@ func NewService(opts ...Option) Service {
|
|||
func (s *service) parse(content []byte) (gast.Node, error) {
|
||||
reader := text.NewReader(content)
|
||||
doc := s.md.Parser().Parse(reader)
|
||||
if masked := maskInvalidLinkReferenceDefinitions(doc, content); masked != nil {
|
||||
doc = s.md.Parser().Parse(text.NewReader(masked))
|
||||
}
|
||||
return doc, nil
|
||||
}
|
||||
|
||||
|
|
@ -139,6 +147,17 @@ func asMemoTagNode(n gast.Node) (*mast.TagNode, bool) {
|
|||
return tagNode, true
|
||||
}
|
||||
|
||||
func appendTagHierarchy(tags []string, value string) []string {
|
||||
for offset := 0; ; {
|
||||
separator := strings.IndexByte(value[offset:], '/')
|
||||
if separator < 0 {
|
||||
return append(tags, value)
|
||||
}
|
||||
offset += separator + 1
|
||||
tags = append(tags, value[:offset-1])
|
||||
}
|
||||
}
|
||||
|
||||
// ExtractTags returns all #tags found in content.
|
||||
func (s *service) ExtractTags(content []byte) ([]string, error) {
|
||||
root, err := s.parse(content)
|
||||
|
|
@ -147,7 +166,6 @@ func (s *service) ExtractTags(content []byte) ([]string, error) {
|
|||
}
|
||||
|
||||
var tags []string
|
||||
|
||||
// Walk the AST to find tag nodes
|
||||
err = gast.Walk(root, func(n gast.Node, entering bool) (gast.WalkStatus, error) {
|
||||
if !entering {
|
||||
|
|
@ -155,7 +173,7 @@ func (s *service) ExtractTags(content []byte) ([]string, error) {
|
|||
}
|
||||
|
||||
if tagNode, ok := asMemoTagNode(n); ok {
|
||||
tags = append(tags, string(tagNode.Tag))
|
||||
tags = appendTagHierarchy(tags, string(tagNode.Tag))
|
||||
}
|
||||
|
||||
return gast.WalkContinue, nil
|
||||
|
|
@ -184,6 +202,14 @@ func extractTextFromNode(n gast.Node, source []byte, buf *strings.Builder) {
|
|||
buf.Write(textNode.Segment.Value(source))
|
||||
return
|
||||
}
|
||||
if mathNode, ok := n.(*mast.InlineMathNode); ok {
|
||||
buf.Write(mathNode.Source)
|
||||
return
|
||||
}
|
||||
if emailNode, ok := n.(*mast.GFMEmailNode); ok {
|
||||
buf.Write(emailNode.Address)
|
||||
return
|
||||
}
|
||||
for child := n.FirstChild(); child != nil; child = child.NextSibling() {
|
||||
extractTextFromNode(child, source, buf)
|
||||
}
|
||||
|
|
@ -213,7 +239,7 @@ func (s *service) ExtractProperties(content []byte) (*storepb.MemoPayload_Proper
|
|||
}
|
||||
|
||||
switch n.Kind() {
|
||||
case gast.KindLink:
|
||||
case gast.KindLink, gast.KindAutoLink, mast.KindGFMEmail:
|
||||
prop.HasLink = true
|
||||
|
||||
case gast.KindCodeBlock, gast.KindFencedCodeBlock, gast.KindCodeSpan:
|
||||
|
|
@ -253,8 +279,13 @@ func (s *service) RenderMarkdown(content []byte) (string, error) {
|
|||
|
||||
// RenderHTML renders markdown content to HTML using goldmark's built-in HTML renderer.
|
||||
func (s *service) RenderHTML(content []byte) (string, error) {
|
||||
root, err := s.parse(content)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := s.md.Convert(content, &buf); err != nil {
|
||||
if err := s.md.Renderer().Render(&buf, content, root); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return buf.String(), nil
|
||||
|
|
@ -316,8 +347,19 @@ func (s *service) GenerateSnippet(content []byte, maxLength int) (string, error)
|
|||
buf.Write(node.URL(content))
|
||||
return gast.WalkSkipChildren, nil
|
||||
case *mast.TagNode:
|
||||
buf.WriteByte('#')
|
||||
buf.Write(node.Tag)
|
||||
if len(node.Source) > 0 {
|
||||
buf.Write(node.Source)
|
||||
} else {
|
||||
buf.WriteByte('#')
|
||||
buf.Write(node.Tag)
|
||||
}
|
||||
case *mast.GFMEmailNode:
|
||||
buf.Write(node.Address)
|
||||
case *mast.InlineMathNode:
|
||||
buf.Write(node.Source)
|
||||
case *mast.BlockMathNode:
|
||||
buf.Write(node.Source)
|
||||
return gast.WalkSkipChildren, nil
|
||||
default:
|
||||
// Ignore other node types.
|
||||
}
|
||||
|
|
@ -366,7 +408,6 @@ func (s *service) ExtractAll(content []byte) (*ExtractedData, error) {
|
|||
}
|
||||
|
||||
firstBlockChecked := false
|
||||
|
||||
// Single walk to collect all data
|
||||
err = gast.Walk(root, func(n gast.Node, entering bool) (gast.WalkStatus, error) {
|
||||
if !entering {
|
||||
|
|
@ -374,7 +415,7 @@ func (s *service) ExtractAll(content []byte) (*ExtractedData, error) {
|
|||
}
|
||||
|
||||
if tagNode, ok := asMemoTagNode(n); ok {
|
||||
data.Tags = append(data.Tags, string(tagNode.Tag))
|
||||
data.Tags = appendTagHierarchy(data.Tags, string(tagNode.Tag))
|
||||
}
|
||||
if mentionNode, ok := n.(*mast.MentionNode); ok {
|
||||
data.Mentions = append(data.Mentions, strings.ToLower(string(mentionNode.Username)))
|
||||
|
|
@ -390,7 +431,7 @@ func (s *service) ExtractAll(content []byte) (*ExtractedData, error) {
|
|||
|
||||
// Extract properties based on node kind
|
||||
switch n.Kind() {
|
||||
case gast.KindLink:
|
||||
case gast.KindLink, gast.KindAutoLink, mast.KindGFMEmail:
|
||||
data.Property.HasLink = true
|
||||
|
||||
case gast.KindCodeBlock, gast.KindFencedCodeBlock, gast.KindCodeSpan:
|
||||
|
|
@ -428,15 +469,19 @@ func (s *service) RenameTag(content []byte, oldTag, newTag string) (string, erro
|
|||
return "", err
|
||||
}
|
||||
|
||||
// Walk the AST to find and rename tag nodes
|
||||
type sourceRange struct {
|
||||
start int
|
||||
end int
|
||||
}
|
||||
var ranges []sourceRange
|
||||
err = gast.Walk(root, func(n gast.Node, entering bool) (gast.WalkStatus, error) {
|
||||
if !entering {
|
||||
return gast.WalkContinue, nil
|
||||
}
|
||||
|
||||
if tagNode, ok := asMemoTagNode(n); ok {
|
||||
if string(tagNode.Tag) == oldTag {
|
||||
tagNode.Tag = []byte(newTag)
|
||||
if string(tagNode.Tag) == oldTag && len(tagNode.Source) > 0 {
|
||||
ranges = append(ranges, sourceRange{start: tagNode.Pos(), end: tagNode.Pos() + len(tagNode.Source)})
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -447,9 +492,18 @@ func (s *service) RenameTag(content []byte, oldTag, newTag string) (string, erro
|
|||
return "", err
|
||||
}
|
||||
|
||||
// Render back to markdown using the already-parsed AST
|
||||
mdRenderer := renderer.NewMarkdownRenderer()
|
||||
return mdRenderer.Render(root, content), nil
|
||||
slices.SortFunc(ranges, func(left, right sourceRange) int { return cmp.Compare(left.start, right.start) })
|
||||
var output bytes.Buffer
|
||||
output.Grow(len(content))
|
||||
cursor := 0
|
||||
for _, sourceRange := range ranges {
|
||||
output.Write(content[cursor:sourceRange.start])
|
||||
output.WriteByte('#')
|
||||
output.WriteString(newTag)
|
||||
cursor = sourceRange.end
|
||||
}
|
||||
output.Write(content[cursor:])
|
||||
return output.String(), nil
|
||||
}
|
||||
|
||||
// uniquePreserveCase returns unique strings from input while preserving case.
|
||||
|
|
|
|||
|
|
@ -498,7 +498,7 @@ func TestExtractTags(t *testing.T) {
|
|||
name: "hierarchical tag with Chinese",
|
||||
content: "#work/测试/项目",
|
||||
withExt: true,
|
||||
expected: []string{"work/测试/项目"},
|
||||
expected: []string{"work", "work/测试", "work/测试/项目"},
|
||||
},
|
||||
}
|
||||
|
||||
|
|
@ -518,6 +518,327 @@ func TestExtractTags(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestExtractTagsMemosTagV1(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
tests := []struct {
|
||||
name string
|
||||
content string
|
||||
expected []string
|
||||
}{
|
||||
{name: "adjacent introducers", content: "#first#second", expected: []string{"first", "second"}},
|
||||
{name: "failed introducer does not hide next", content: "##tag", expected: []string{"tag"}},
|
||||
{name: "ATX heading marker", content: "## tag", expected: []string{}},
|
||||
{name: "number sign keycap", content: "#️⃣", expected: []string{}},
|
||||
{name: "number sign keycap value", content: "##️⃣", expected: []string{"#️⃣"}},
|
||||
{name: "number sign keycap continuation", content: "#first#️⃣", expected: []string{"first#️⃣"}},
|
||||
{name: "exact identity", content: "#Work #work", expected: []string{"Work", "work"}},
|
||||
{name: "no normalization", content: "#café #cafe\u0301", expected: []string{"café", "cafe\u0301"}},
|
||||
{name: "ignored spellings deduplicate", content: "#AB #A\u200dB #\u0301AB", expected: []string{"AB"}},
|
||||
{name: "hierarchy expansion", content: "#book/fiction/history", expected: []string{"book", "book/fiction", "book/fiction/history"}},
|
||||
{name: "hierarchy exact dedupe", content: "#book/fiction #book", expected: []string{"book", "book/fiction"}},
|
||||
{name: "literal ampersand", content: "#R&D", expected: []string{"R&D"}},
|
||||
{name: "named character reference boundary", content: "#R&D", expected: []string{"R"}},
|
||||
{name: "character reference requires a real opener", content: "~#a!&&)*", expected: []string{"a"}},
|
||||
{name: "unknown character reference stays literal", content: "#R&bogus;D #Q&&bogus;D", expected: []string{"R&bogus", "Q"}},
|
||||
{name: "numeric character reference introducer", content: "#tag #other", expected: []string{}},
|
||||
{name: "named character reference introducer", content: "#tag", expected: []string{}},
|
||||
{name: "escaped introducer and body", content: "\\#tag #foo\\+bar", expected: []string{"foo"}},
|
||||
{name: "code contexts", content: "`#inline`\n\n```text\n#fenced\n```\n\n #indented", expected: []string{}},
|
||||
{name: "blockquote tab paragraph", content: ">\t #tag", expected: []string{"tag"}},
|
||||
{name: "trailing spaces before tag line", content: "before \n#tag", expected: []string{"tag"}},
|
||||
{name: "link and image contexts", content: "[#label](https://example.com/#destination) ", expected: []string{}},
|
||||
{name: "invalid link definition stays text", content: "[#use][bad]\n\n[bad]:(", expected: []string{"use"}},
|
||||
{name: "invalid link definition label stays text", content: "[#label]:(", expected: []string{"label"}},
|
||||
{name: "email in invalid link definition stays opaque", content: "[#foo@example.com]:(", expected: []string{}},
|
||||
{name: "code in invalid link definition stays opaque", content: "[`#code`]:(", expected: []string{}},
|
||||
{name: "URL in invalid link definition stays opaque", content: "[ https://example.com/#url]:(", expected: []string{}},
|
||||
{name: "balanced bare reference remains a link", content: "[#hidden][valid]\n\n[valid]: path(foo)", expected: []string{}},
|
||||
{name: "escaped bare destination parenthesis remains a link", content: "[#hidden][valid]\n\n[valid]: path\\(", expected: []string{}},
|
||||
{name: "angle destination parenthesis remains a link", content: "[#hidden][valid]\n\n[valid]: <(>", expected: []string{}},
|
||||
{
|
||||
name: "angle autolinks stay opaque",
|
||||
content: "<https://localhost/#local> <ftp://example.com/#ftp> <custom:foo#custom> <https://foo.example_/#underscore>",
|
||||
expected: []string{},
|
||||
},
|
||||
{name: "literal URL context", content: "https://example.com/path#url /path#plain", expected: []string{"plain"}},
|
||||
{
|
||||
name: "non GFM URL schemes and forms stay text",
|
||||
content: "ftp://example.com/#ftp HTTP://example.com/#upper https://localhost/#local",
|
||||
expected: []string{"ftp", "upper", "local"},
|
||||
},
|
||||
{
|
||||
name: "written GFM URL domains and paths",
|
||||
content: "https://example.COM/#hidden www.example.123/#also-hidden https://a_b.foo.example/路径#still-hidden https://foo_bar.example/#visible",
|
||||
expected: []string{"visible"},
|
||||
},
|
||||
{
|
||||
name: "written GFM URL Unicode domains",
|
||||
content: "https://點看.com/#hidden www.點看.com/#also-hidden https://a_b.點看.com/#still-hidden " +
|
||||
"https://foo_bar.點看/#visible https://點看.com_/#suffix hellohttps://點看.com/#joined",
|
||||
expected: []string{"visible", "suffix", "joined"},
|
||||
},
|
||||
{
|
||||
name: "written GFM URL protocol and www boundaries",
|
||||
content: "1https://點看.com/#digit .https://點看.com/#punctuation ]www.點看.com/#www " +
|
||||
"中https://點看.com/#unicode hellohttps://點看.com/#joined " +
|
||||
"[https://點看.com/#unbalanced\n\n[www.點看.com/#unbalanced-www\n\n" +
|
||||
`\[https://點看.com/#escaped` + "\n\n" + `\[www.點看.com/#escaped-www`,
|
||||
expected: []string{
|
||||
"digit", "punctuation", "www", "unicode", "joined", "unbalanced", "unbalanced-www", "escaped", "escaped-www",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "written GFM URL domain validation",
|
||||
content: "http://example.com#hidden http://example.com./#also-hidden https://foo.example_/#visible\n" +
|
||||
"-\thttps://foo.example_/#padded\n\n(https://foo.example_/#parenthesized) *https://foo.example_/#emphasized*",
|
||||
expected: []string{"visible", "padded", "parenthesized", "emphasized"},
|
||||
},
|
||||
{name: "initial BOM before written GFM URL", content: "\ufeffhttps://example.com/#hidden #shown", expected: []string{"shown"}},
|
||||
{name: "second initial BOM is ordinary text", content: "\ufeff\ufeffhttps://example.com/#shown", expected: []string{"shown"}},
|
||||
{name: "mid-document BOM does not start written GFM URL", content: "before\n\n\ufeffhttps://example.com/#shown", expected: []string{"shown"}},
|
||||
{name: "written URL inside unresolved bracket text", content: "[text https://點看.com/#inside [more https://example.com/#ascii", expected: []string{}},
|
||||
{
|
||||
name: "written URL inside closed unresolved bracket text",
|
||||
content: "[ https://example.com/#hidden] [x https://example.com/#also-hidden] ![ https://example.com/#image-hidden]",
|
||||
expected: []string{},
|
||||
},
|
||||
{
|
||||
name: "separator characters end written URL",
|
||||
content: "https://點看.com/\u00a0#nbsp https://點看.com/\u2003#em-space https://點看.com/\u000b#vertical-tab " +
|
||||
"https://點看.com/\u2028#line-separator https://點看.com/\u2029#paragraph-separator https://點看.com/\u0085#nel-hidden",
|
||||
expected: []string{"nbsp", "em-space", "vertical-tab", "line-separator", "paragraph-separator"},
|
||||
},
|
||||
{
|
||||
name: "written GFM email boundaries",
|
||||
content: "foo#mail@example.com foo!#second@example.com <foo#hidden@example.com> foo@example.com/#after",
|
||||
expected: []string{"after"},
|
||||
},
|
||||
{name: "invalid GFM email local part", content: "#foo!bar@example.com", expected: []string{"foo"}},
|
||||
{name: "email after tag hierarchy separator", content: "#foo/bar@example.com", expected: []string{"foo"}},
|
||||
{name: "GFM email after Unicode tag", content: "#中mail@example.com", expected: []string{"中"}},
|
||||
{name: "email after padded list tag", content: "-\t#foo/bar@example.com", expected: []string{"foo"}},
|
||||
{name: "invalid GFM email domain suffix", content: "#foo/bar@example.com_", expected: []string{"foo", "foo/bar"}},
|
||||
{name: "escaped punctuation is decoded in GFM email autolinks", content: `#foo\+bar@example.com #next\_item@example.com`, expected: []string{}},
|
||||
{name: "numeric entity joins GFM email", content: `#foo.bar@example.com #foo@example.com`, expected: []string{}},
|
||||
{name: "named entity joins GFM email", content: `#foo.bar@example.com`, expected: []string{}},
|
||||
{name: "unknown entity does not join GFM email", content: `#foo&bogus;bar@example.com`, expected: []string{"foo&bogus"}},
|
||||
{name: "email inside emphasis", content: "_foo@example.com #tag_", expected: []string{"tag"}},
|
||||
{name: "raw HTML syntax", content: "<span data-tag=\"#attribute\">#text</span>", expected: []string{"text"}},
|
||||
{name: "inline math", content: "$#math$ #plain", expected: []string{"plain"}},
|
||||
{name: "inline math exact closing run", content: "$#one$$ #two$ #plain", expected: []string{"plain"}},
|
||||
{name: "inline math does not reopen within a dollar run", content: "$$#math$ #plain", expected: []string{"math", "plain"}},
|
||||
{name: "inline math after escaped dollar", content: "\\$$#math$ #plain", expected: []string{"plain"}},
|
||||
{name: "display math", content: "$$\n#math\n$$\n\n#plain", expected: []string{"plain"}},
|
||||
{name: "unclosed math flow", content: "$$meta\n#math", expected: []string{}},
|
||||
{name: "unclosed math flow across blocks", content: "$$meta\n\n#math", expected: []string{}},
|
||||
{name: "unclosed math flow in quote", content: "> $$meta\n> #math", expected: []string{}},
|
||||
{name: "flow math stays inside quote", content: "> $$\n> #math\noutside #plain", expected: []string{"plain"}},
|
||||
{name: "flow math spans quote blocks", content: "> $$\n> #math\n>\n> #still-math\n> $$\noutside #plain", expected: []string{"plain"}},
|
||||
{name: "flow math stays inside list item", content: "- $$\n #math\n#outside", expected: []string{"outside"}},
|
||||
{name: "flow math spans list item blank lines", content: "- $$\n \n #math\n $$\n\n#outside", expected: []string{"outside"}},
|
||||
{name: "dollar in flow metadata stays text", content: "$$meta$x\n#plain", expected: []string{"plain"}},
|
||||
{name: "math fence inside code is inert", content: "```text\n$$\n```\n\n#plain", expected: []string{"plain"}},
|
||||
{name: "unmatched inline dollar", content: "text $ #plain", expected: []string{"plain"}},
|
||||
{name: "formatted text", content: "**#strong** ~~#strike~~", expected: []string{"strong", "strike"}},
|
||||
{name: "emphasis is a hard boundary", content: "#_foo_", expected: []string{}},
|
||||
{name: "unmatched underscore", content: "#_", expected: []string{"_"}},
|
||||
{name: "intraword underscore", content: "#foo_bar", expected: []string{"foo_bar"}},
|
||||
{name: "no length limit", content: "#" + strings.Repeat("a", 101), expected: []string{strings.Repeat("a", 101)}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
tags, err := svc.ExtractTags([]byte(tt.content))
|
||||
require.NoError(t, err)
|
||||
if len(tt.expected) == 0 {
|
||||
assert.Empty(t, tags)
|
||||
} else {
|
||||
assert.Equal(t, tt.expected, tags)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractAllExpandsTagHierarchy(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
data, err := svc.ExtractAll([]byte("#book/fiction/history #book"))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"book", "book/fiction", "book/fiction/history"}, data.Tags)
|
||||
}
|
||||
|
||||
func TestExtractAllResolvesGFMEmailBeforeMention(t *testing.T) {
|
||||
svc := NewService(WithTagExtension(), WithMentionExtension())
|
||||
data, err := svc.ExtractAll([]byte("#foo/bar_baz@example.com #next/item_@example.com @alice"))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"foo", "next"}, data.Tags)
|
||||
assert.Equal(t, []string{"alice"}, data.Mentions)
|
||||
assert.True(t, data.Property.HasLink)
|
||||
}
|
||||
|
||||
func TestInvalidLinkDefinitionsRetainInlineContexts(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
data, err := svc.ExtractAll([]byte("[#foo@example.com]:(\n\n[`#code`]:("))
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, data.Tags)
|
||||
assert.True(t, data.Property.HasLink)
|
||||
assert.True(t, data.Property.HasCode)
|
||||
}
|
||||
|
||||
func TestTagSourceSpelling(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
content := "#A\u200dB #\u0301foo"
|
||||
|
||||
rendered, err := svc.RenderMarkdown([]byte(content))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, content, rendered)
|
||||
|
||||
snippet, err := svc.GenerateSnippet([]byte(content), 100)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, content, snippet)
|
||||
|
||||
renamed, err := svc.RenameTag([]byte(content), "AB", "done")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "#done #\u0301foo", renamed)
|
||||
}
|
||||
|
||||
func TestRenderHTMLPreservesTagSourceSpelling(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
html, err := svc.RenderHTML([]byte("#R&D #A\u200dB"))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "<p>#R&D #A\u200dB</p>\n", html)
|
||||
}
|
||||
|
||||
func TestRenderHTMLRejectsUnclosedReferenceDestination(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
html, err := svc.RenderHTML([]byte("[#use][bad]\n\n[bad]:("))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "<p>[#use][bad]</p>\n<p>[bad]:(</p>\n", html)
|
||||
}
|
||||
|
||||
func TestRenderHTMLRecognizesGFMEmails(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
tests := []struct {
|
||||
content string
|
||||
expected string
|
||||
}{
|
||||
{
|
||||
content: "foo#mail@example.com",
|
||||
expected: "<p>foo#<a href=\"mailto:mail@example.com\">mail@example.com</a></p>\n",
|
||||
},
|
||||
{
|
||||
content: "#foo/bar_baz@example.com",
|
||||
expected: "<p>#foo/<a href=\"mailto:bar_baz@example.com\">bar_baz@example.com</a></p>\n",
|
||||
},
|
||||
{
|
||||
content: "_foo@example.com #tag_",
|
||||
expected: "<p><em><a href=\"mailto:foo@example.com\">foo@example.com</a> #tag</em></p>\n",
|
||||
},
|
||||
{
|
||||
content: "foo@bar.com@baz.example",
|
||||
expected: "<p><a href=\"mailto:foo@bar.com\">foo@bar.com</a>@baz.example</p>\n",
|
||||
},
|
||||
{
|
||||
content: "foo@bar.com+abc@def.com",
|
||||
expected: "<p><a href=\"mailto:foo@bar.com\">foo@bar.com</a><a href=\"mailto:+abc@def.com\">+abc@def.com</a></p>\n",
|
||||
},
|
||||
{
|
||||
content: `#foo\+bar@example.com`,
|
||||
expected: "<p>#<a href=\"mailto:foo+bar@example.com\">foo+bar@example.com</a></p>\n",
|
||||
},
|
||||
{
|
||||
content: `#foo\@example\.com`,
|
||||
expected: "<p>#<a href=\"mailto:foo@example.com\">foo@example.com</a></p>\n",
|
||||
},
|
||||
{
|
||||
content: `#foo.bar@example.com`,
|
||||
expected: "<p>#<a href=\"mailto:foo.bar@example.com\">foo.bar@example.com</a></p>\n",
|
||||
},
|
||||
{
|
||||
content: `#foo.bar@example.com`,
|
||||
expected: "<p>#<a href=\"mailto:foo.bar@example.com\">foo.bar@example.com</a></p>\n",
|
||||
},
|
||||
{
|
||||
content: `#foo@example.com`,
|
||||
expected: "<p>#<a href=\"mailto:foo@example.com\">foo@example.com</a></p>\n",
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
html, err := svc.RenderHTML([]byte(test.content))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, test.expected, html)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderHTMLRecognizesGFMEmailWithoutTagExtension(t *testing.T) {
|
||||
svc := NewService()
|
||||
html, err := svc.RenderHTML([]byte("mail@example.com"))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "<p><a href=\"mailto:mail@example.com\">mail@example.com</a></p>\n", html)
|
||||
}
|
||||
|
||||
func TestRenderHTMLRecognizesGFMURLsInsideUnresolvedBracketText(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
for _, content := range []string{
|
||||
"[ https://example.com/#hidden]",
|
||||
"[x https://example.com/#hidden]",
|
||||
"![ https://example.com/#hidden]",
|
||||
"[text https://example.com/#hidden",
|
||||
} {
|
||||
html, err := svc.RenderHTML([]byte(content))
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, html, "<a href=\"https://example.com/#hidden")
|
||||
|
||||
data, err := svc.ExtractAll([]byte(content))
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, data.Tags)
|
||||
assert.True(t, data.Property.HasLink)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEscapedGFMEmailPreservesSource(t *testing.T) {
|
||||
svc := NewService(WithTagExtension(), WithMentionExtension())
|
||||
content := `#foo\+bar@example.com #next\_item@example.com #foo\@example\.com`
|
||||
|
||||
rendered, err := svc.RenderMarkdown([]byte(content))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, content, rendered)
|
||||
|
||||
data, err := svc.ExtractAll([]byte(content))
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, data.Tags)
|
||||
assert.Empty(t, data.Mentions)
|
||||
assert.True(t, data.Property.HasLink)
|
||||
}
|
||||
|
||||
func TestMathRenderingPreservesLiteralSource(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
for _, content := range []string{
|
||||
"$x < y$\n\n$$meta\nx < y\n$$",
|
||||
"> $$meta\n> x < y\n> $$",
|
||||
"- $$meta\n x < y\n $$",
|
||||
"- $$meta\n \n x < y\n $$",
|
||||
"- $$meta\n x < y\n $$\n- next",
|
||||
"10. $$meta\n x < y\n $$",
|
||||
} {
|
||||
rendered, err := svc.RenderMarkdown([]byte(content))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, content, rendered)
|
||||
}
|
||||
|
||||
html, err := svc.RenderHTML([]byte("$x < y$\n\n$$meta\nx < y\n$$"))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "<p>$x < y$</p>\n$$meta\nx < y\n$$", html)
|
||||
}
|
||||
|
||||
func TestRenderMarkdownPreservesLineBreakAfterTag(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
for _, content := range []string{"#tag\nnext", "#tag \nnext"} {
|
||||
rendered, err := svc.RenderMarkdown([]byte(content))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, content, rendered)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenameTagSkipsTagsInsideLinks(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
|
||||
|
|
@ -531,6 +852,28 @@ func TestRenameTagSkipsTagsInsideLinks(t *testing.T) {
|
|||
assert.Equal(t, "[release #notes](https://example.com/releases#release-notes)\n\nOutside #done", result)
|
||||
}
|
||||
|
||||
func TestRenameTagOnlyChangesRecognizedSourceSpans(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
for _, content := range []string{
|
||||
"<span>#old</span>",
|
||||
"<https://example.com/#x> user@example.com #old",
|
||||
"#old\n$$\nx\n$$\nnext",
|
||||
} {
|
||||
result, err := svc.RenameTag([]byte(content), "old", "new")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, strings.ReplaceAll(content, "#old", "#new"), result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenameTagReplacesEveryExactSourceSpan(t *testing.T) {
|
||||
svc := NewService(WithTagExtension())
|
||||
content := "#old #o\u200dld #old#old [#old](url) https://example.com/#old #Old"
|
||||
|
||||
result, err := svc.RenameTag([]byte(content), "old", "new")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "#new #new #new#new [#old](url) https://example.com/#old #Old", result)
|
||||
}
|
||||
|
||||
func TestUniquePreserveCase(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
|
|
|||
140
internal/markdown/parser/gfm_email.go
Normal file
140
internal/markdown/parser/gfm_email.go
Normal file
|
|
@ -0,0 +1,140 @@
|
|||
package parser
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"regexp"
|
||||
|
||||
"github.com/yuin/goldmark/util"
|
||||
)
|
||||
|
||||
var gfmEmailDomainPattern = regexp.MustCompile(`^@(?:[A-Za-z0-9_-]+\.)+[A-Za-z0-9_-]*[A-Za-z0-9]`)
|
||||
|
||||
// GFMEmailMatch is one extended GFM email autolink source range.
|
||||
type GFMEmailMatch struct {
|
||||
Start int
|
||||
End int
|
||||
Address []byte
|
||||
}
|
||||
|
||||
type gfmEmailProjection struct {
|
||||
value []byte
|
||||
starts []int
|
||||
ends []int
|
||||
}
|
||||
|
||||
func projectGFMEmailText(source []byte) gfmEmailProjection {
|
||||
projection := gfmEmailProjection{
|
||||
value: make([]byte, 0, len(source)),
|
||||
starts: make([]int, 0, len(source)),
|
||||
ends: make([]int, 0, len(source)),
|
||||
}
|
||||
for cursor := 0; cursor < len(source); {
|
||||
if source[cursor] == '\\' && cursor+1 < len(source) && util.IsPunct(source[cursor+1]) {
|
||||
projection.value = append(projection.value, source[cursor+1])
|
||||
projection.starts = append(projection.starts, cursor)
|
||||
projection.ends = append(projection.ends, cursor+2)
|
||||
cursor += 2
|
||||
continue
|
||||
}
|
||||
if referenceLength := characterReferenceLength(source[cursor:]); referenceLength > 0 {
|
||||
end := cursor + referenceLength
|
||||
decoded := util.ResolveEntityNames(util.ResolveNumericReferences(source[cursor:end]))
|
||||
projection.value = append(projection.value, decoded...)
|
||||
for range len(decoded) {
|
||||
projection.starts = append(projection.starts, cursor)
|
||||
projection.ends = append(projection.ends, end)
|
||||
}
|
||||
cursor = end
|
||||
continue
|
||||
}
|
||||
projection.value = append(projection.value, source[cursor])
|
||||
projection.starts = append(projection.starts, cursor)
|
||||
projection.ends = append(projection.ends, cursor+1)
|
||||
cursor++
|
||||
}
|
||||
return projection
|
||||
}
|
||||
|
||||
// FindGFMEmailMatches finds extended GFM email autolinks in one literal text node.
|
||||
func FindGFMEmailMatches(source []byte) []GFMEmailMatch {
|
||||
projection := projectGFMEmailText(source)
|
||||
var matches []GFMEmailMatch
|
||||
for cursor := 0; cursor < len(projection.value); {
|
||||
relativeAt := bytes.IndexByte(projection.value[cursor:], '@')
|
||||
if relativeAt < 0 {
|
||||
break
|
||||
}
|
||||
at := cursor + relativeAt
|
||||
start, end, ok := matchProjectedGFMEmailAt(projection.value, at, cursor, len(projection.value))
|
||||
if !ok {
|
||||
cursor = at + 1
|
||||
continue
|
||||
}
|
||||
matches = append(matches, GFMEmailMatch{
|
||||
Start: projection.starts[start],
|
||||
End: projection.ends[end-1],
|
||||
Address: append([]byte(nil), projection.value[start:end]...),
|
||||
})
|
||||
cursor = end
|
||||
}
|
||||
return matches
|
||||
}
|
||||
|
||||
// MatchGFMEmailAt matches an extended GFM email around the @ at the given byte offset.
|
||||
func MatchGFMEmailAt(source []byte, at, lowerBound, upperBound int) (int, int, bool) {
|
||||
if lowerBound < 0 || at < lowerBound || at >= upperBound || upperBound > len(source) || source[at] != '@' {
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
projection := projectGFMEmailText(source[lowerBound:upperBound])
|
||||
relativeAt := at - lowerBound
|
||||
projectedAt := -1
|
||||
for index, start := range projection.starts {
|
||||
if start == relativeAt && projection.value[index] == '@' {
|
||||
projectedAt = index
|
||||
break
|
||||
}
|
||||
}
|
||||
if projectedAt < 0 {
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
start, end, ok := matchProjectedGFMEmailAt(projection.value, projectedAt, 0, len(projection.value))
|
||||
if !ok {
|
||||
return 0, 0, false
|
||||
}
|
||||
return lowerBound + projection.starts[start], lowerBound + projection.ends[end-1], true
|
||||
}
|
||||
|
||||
func matchProjectedGFMEmailAt(source []byte, at, lowerBound, upperBound int) (int, int, bool) {
|
||||
if lowerBound < 0 || at < lowerBound || at >= upperBound || upperBound > len(source) || source[at] != '@' {
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
start := at
|
||||
for start > lowerBound {
|
||||
if !isGFMEmailLocalByte(source[start-1]) {
|
||||
break
|
||||
}
|
||||
start--
|
||||
}
|
||||
if start == at {
|
||||
return 0, 0, false
|
||||
}
|
||||
domain := gfmEmailDomainPattern.FindIndex(source[at:upperBound])
|
||||
if domain == nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
end := at + domain[1]
|
||||
if end < upperBound && (source[end] == '-' || source[end] == '_') {
|
||||
return 0, 0, false
|
||||
}
|
||||
return start, end, true
|
||||
}
|
||||
|
||||
func isGFMEmailLocalByte(value byte) bool {
|
||||
return value >= 'a' && value <= 'z' ||
|
||||
value >= 'A' && value <= 'Z' ||
|
||||
value >= '0' && value <= '9' ||
|
||||
value == '.' || value == '_' || value == '+' || value == '-'
|
||||
}
|
||||
62
internal/markdown/parser/gfm_email_test.go
Normal file
62
internal/markdown/parser/gfm_email_test.go
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
package parser
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func emailMatch(start, end int, address string) GFMEmailMatch {
|
||||
return GFMEmailMatch{Start: start, End: end, Address: []byte(address)}
|
||||
}
|
||||
|
||||
func TestFindGFMEmailMatches(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
source string
|
||||
expected []GFMEmailMatch
|
||||
}{
|
||||
{name: "basic", source: "mail@example.com", expected: []GFMEmailMatch{emailMatch(0, 16, "mail@example.com")}},
|
||||
{name: "slash starts a new local part", source: "#foo/bar_baz@example.com", expected: []GFMEmailMatch{emailMatch(5, 24, "bar_baz@example.com")}},
|
||||
{name: "punctuation starts a new local part", source: "#foo!bar@example.com", expected: []GFMEmailMatch{emailMatch(5, 20, "bar@example.com")}},
|
||||
{name: "hash starts a text-node email", source: "#foo@example.com", expected: []GFMEmailMatch{emailMatch(1, 16, "foo@example.com")}},
|
||||
{name: "Unicode starts a new local part", source: "#中foo@example.com", expected: []GFMEmailMatch{emailMatch(4, 19, "foo@example.com")}},
|
||||
{name: "emoji starts a new local part", source: "😀foo@example.com", expected: []GFMEmailMatch{emailMatch(4, 19, "foo@example.com")}},
|
||||
{name: "trailing dot excluded", source: "mail@example.com.", expected: []GFMEmailMatch{emailMatch(0, 16, "mail@example.com")}},
|
||||
{name: "invalid domain suffix", source: "mail@example.com_", expected: nil},
|
||||
{name: "multiple", source: "a@b.co c@d.io", expected: []GFMEmailMatch{emailMatch(0, 6, "a@b.co"), emailMatch(7, 13, "c@d.io")}},
|
||||
{name: "first email wins", source: "foo@bar.com@baz.example", expected: []GFMEmailMatch{emailMatch(0, 11, "foo@bar.com")}},
|
||||
{
|
||||
name: "adjacent emails do not overlap",
|
||||
source: "foo@bar.com+abc@def.com",
|
||||
expected: []GFMEmailMatch{emailMatch(0, 11, "foo@bar.com"), emailMatch(11, 23, "+abc@def.com")},
|
||||
},
|
||||
{name: "escaped local plus", source: `#foo\+bar@example.com`, expected: []GFMEmailMatch{emailMatch(1, 21, "foo+bar@example.com")}},
|
||||
{name: "escaped local underscore", source: `#next\_item@example.com`, expected: []GFMEmailMatch{emailMatch(1, 23, "next_item@example.com")}},
|
||||
{name: "escaped at and domain dot", source: `#foo\@example\.com`, expected: []GFMEmailMatch{emailMatch(1, 18, "foo@example.com")}},
|
||||
{name: "numeric dot reference", source: `#foo.bar@example.com`, expected: []GFMEmailMatch{emailMatch(1, 24, "foo.bar@example.com")}},
|
||||
{name: "named dot reference", source: `#foo.bar@example.com`, expected: []GFMEmailMatch{emailMatch(1, 27, "foo.bar@example.com")}},
|
||||
{name: "numeric at reference", source: `#foo@example.com`, expected: []GFMEmailMatch{emailMatch(1, 20, "foo@example.com")}},
|
||||
{name: "unknown reference stays literal", source: `#foo&bogus;bar@example.com`, expected: []GFMEmailMatch{emailMatch(11, 26, "bar@example.com")}},
|
||||
{name: "allowed delimiter", source: "(mail@example.com", expected: []GFMEmailMatch{emailMatch(1, 17, "mail@example.com")}},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
assert.Equal(t, test.expected, FindGFMEmailMatches([]byte(test.source)))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchGFMEmailAtPreservesEscapedSourceRange(t *testing.T) {
|
||||
source := []byte("first@example.com\n#foo\\+bar@example.com")
|
||||
lowerBound := bytes.IndexByte(source, '\n') + 1
|
||||
at := bytes.LastIndexByte(source, '@')
|
||||
start, end, ok := MatchGFMEmailAt(source, at, lowerBound, len(source))
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, lowerBound+1, start)
|
||||
assert.Equal(t, len(source), end)
|
||||
assert.Equal(t, `foo\+bar@example.com`, string(source[start:end]))
|
||||
}
|
||||
148
internal/markdown/parser/math.go
Normal file
148
internal/markdown/parser/math.go
Normal file
|
|
@ -0,0 +1,148 @@
|
|||
package parser
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
|
||||
gast "github.com/yuin/goldmark/ast"
|
||||
"github.com/yuin/goldmark/parser"
|
||||
"github.com/yuin/goldmark/text"
|
||||
"github.com/yuin/goldmark/util"
|
||||
|
||||
mast "github.com/usememos/memos/internal/markdown/ast"
|
||||
)
|
||||
|
||||
type inlineMathParser struct{}
|
||||
|
||||
// NewInlineMathParser creates a parser for exact dollar-run inline math.
|
||||
func NewInlineMathParser() parser.InlineParser {
|
||||
return &inlineMathParser{}
|
||||
}
|
||||
|
||||
func (*inlineMathParser) Trigger() []byte {
|
||||
return []byte{'$'}
|
||||
}
|
||||
|
||||
func (*inlineMathParser) Parse(_ gast.Node, reader text.Reader, _ parser.Context) gast.Node {
|
||||
line, _ := reader.PeekLine()
|
||||
openingLength := dollarRunLength(line)
|
||||
if openingLength == 0 || hasUnescapedPrecedingDollar(reader) {
|
||||
return nil
|
||||
}
|
||||
|
||||
savedLine, savedPosition := reader.Position()
|
||||
source := append([]byte(nil), line[:openingLength]...)
|
||||
reader.Advance(openingLength)
|
||||
|
||||
for {
|
||||
line, _ = reader.PeekLine()
|
||||
if line == nil {
|
||||
reader.SetPosition(savedLine, savedPosition)
|
||||
return nil
|
||||
}
|
||||
|
||||
for pos := 0; pos < len(line); {
|
||||
if line[pos] != '$' {
|
||||
pos++
|
||||
continue
|
||||
}
|
||||
closingLength := dollarRunLength(line[pos:])
|
||||
if closingLength == openingLength {
|
||||
end := pos + closingLength
|
||||
source = append(source, line[:end]...)
|
||||
reader.Advance(end)
|
||||
return &mast.InlineMathNode{Source: source}
|
||||
}
|
||||
pos += closingLength
|
||||
}
|
||||
|
||||
source = append(source, line...)
|
||||
reader.AdvanceLine()
|
||||
}
|
||||
}
|
||||
|
||||
// hasUnescapedPrecedingDollar prevents retrying within one dollar run while
|
||||
// allowing a new run after an escaped dollar.
|
||||
func hasUnescapedPrecedingDollar(reader text.Reader) bool {
|
||||
if reader.PrecendingCharacter() != '$' {
|
||||
return false
|
||||
}
|
||||
_, position := reader.Position()
|
||||
previous := position.Start - 1
|
||||
if previous < 0 || reader.Source()[previous] != '$' {
|
||||
return true
|
||||
}
|
||||
|
||||
backslashes := 0
|
||||
for pos := previous - 1; pos >= 0 && reader.Source()[pos] == '\\'; pos-- {
|
||||
backslashes++
|
||||
}
|
||||
return backslashes%2 == 0
|
||||
}
|
||||
|
||||
type blockMathParser struct{}
|
||||
|
||||
// NewBlockMathParser creates a parser for dollar-fenced flow math.
|
||||
func NewBlockMathParser() parser.BlockParser {
|
||||
return &blockMathParser{}
|
||||
}
|
||||
|
||||
func (*blockMathParser) Trigger() []byte {
|
||||
return []byte{'$'}
|
||||
}
|
||||
|
||||
func (*blockMathParser) Open(_ gast.Node, reader text.Reader, context parser.Context) (gast.Node, parser.State) {
|
||||
line, _ := reader.PeekLine()
|
||||
pos := context.BlockOffset()
|
||||
if pos < 0 || pos >= len(line) {
|
||||
return nil, parser.NoChildren
|
||||
}
|
||||
|
||||
fenceLength := dollarRunLength(line[pos:])
|
||||
if fenceLength < 2 || bytes.IndexByte(line[pos+fenceLength:], '$') >= 0 {
|
||||
return nil, parser.NoChildren
|
||||
}
|
||||
|
||||
node := mast.NewBlockMathNode(line, fenceLength)
|
||||
reader.AdvanceToEOL()
|
||||
return node, parser.NoChildren
|
||||
}
|
||||
|
||||
func (*blockMathParser) Continue(node gast.Node, reader text.Reader, _ parser.Context) parser.State {
|
||||
mathNode, ok := node.(*mast.BlockMathNode)
|
||||
if !ok {
|
||||
return parser.Close
|
||||
}
|
||||
|
||||
line, _ := reader.PeekLine()
|
||||
width, pos := util.IndentWidth(line, reader.LineOffset())
|
||||
if width < 4 {
|
||||
fenceLength := dollarRunLength(line[pos:])
|
||||
if fenceLength >= mathNode.FenceLength() && util.IsBlank(line[pos+fenceLength:]) {
|
||||
mathNode.AppendSource(line)
|
||||
reader.AdvanceToEOL()
|
||||
return parser.Close
|
||||
}
|
||||
}
|
||||
|
||||
mathNode.AppendSource(line)
|
||||
reader.AdvanceToEOL()
|
||||
return parser.Continue | parser.NoChildren
|
||||
}
|
||||
|
||||
func (*blockMathParser) Close(gast.Node, text.Reader, parser.Context) {}
|
||||
|
||||
func (*blockMathParser) CanInterruptParagraph() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
func (*blockMathParser) CanAcceptIndentedLine() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func dollarRunLength(source []byte) int {
|
||||
length := 0
|
||||
for length < len(source) && source[length] == '$' {
|
||||
length++
|
||||
}
|
||||
return length
|
||||
}
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
package parser
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
|
|
@ -38,10 +39,14 @@ func isMentionBoundary(r rune) bool {
|
|||
|
||||
// Parse parses @mention syntax while avoiding email-address matches.
|
||||
func (*mentionParser) Parse(_ gast.Node, block text.Reader, _ parser.Context) gast.Node {
|
||||
line, _ := block.PeekLine()
|
||||
line, segment := block.PeekLine()
|
||||
if len(line) == 0 || line[0] != '@' {
|
||||
return nil
|
||||
}
|
||||
lineStart := bytes.LastIndexByte(block.Source()[:segment.Start], '\n') + 1
|
||||
if _, _, ok := MatchGFMEmailAt(block.Source(), segment.Start, lineStart, segment.Start+len(line)); ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
prev := block.PrecendingCharacter()
|
||||
if prev != '\n' && !isMentionBoundary(prev) {
|
||||
|
|
|
|||
|
|
@ -1,150 +1,235 @@
|
|||
package parser
|
||||
|
||||
import (
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
gast "github.com/yuin/goldmark/ast"
|
||||
"github.com/yuin/goldmark/parser"
|
||||
"github.com/yuin/goldmark/text"
|
||||
|
||||
mast "github.com/usememos/memos/internal/markdown/ast"
|
||||
"github.com/yuin/goldmark/util"
|
||||
)
|
||||
|
||||
const (
|
||||
// MaxTagLength defines the maximum number of runes allowed in a tag.
|
||||
MaxTagLength = 100
|
||||
)
|
||||
//go:generate go run ./tagdata -output tag_unicode_tables.go
|
||||
|
||||
type tagParser struct{}
|
||||
|
||||
// NewTagParser creates a new inline parser for #tag syntax.
|
||||
func NewTagParser() parser.InlineParser {
|
||||
return &tagParser{}
|
||||
type codePointRange struct {
|
||||
lo rune
|
||||
hi rune
|
||||
}
|
||||
|
||||
// Trigger returns the characters that trigger this parser.
|
||||
func (*tagParser) Trigger() []byte {
|
||||
return []byte{'#'}
|
||||
// TagMatch is one lexical tag match within an eligible literal-source run.
|
||||
type TagMatch struct {
|
||||
// Start is the byte offset of the introducer within the source run.
|
||||
Start int
|
||||
// End is the exclusive byte offset of the recognized source spelling.
|
||||
End int
|
||||
// Value is the emitted direct tag value without the introducer.
|
||||
Value []byte
|
||||
}
|
||||
|
||||
// isValidTagRune checks if a Unicode rune is valid in a tag.
|
||||
// Uses Unicode categories for proper international character support.
|
||||
func isValidTagRune(r rune) bool {
|
||||
// Allow Unicode letters (any script: Latin, CJK, Arabic, Cyrillic, etc.)
|
||||
if unicode.IsLetter(r) {
|
||||
return true
|
||||
func containsCodePoint(ranges []codePointRange, r rune) bool {
|
||||
low, high := 0, len(ranges)
|
||||
for low < high {
|
||||
middle := low + (high-low)/2
|
||||
current := ranges[middle]
|
||||
switch {
|
||||
case r < current.lo:
|
||||
high = middle
|
||||
case r > current.hi:
|
||||
low = middle + 1
|
||||
default:
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// Allow Unicode digits
|
||||
if unicode.IsNumber(r) {
|
||||
return true
|
||||
}
|
||||
|
||||
// Allow emoji and symbols (So category: Symbol, Other)
|
||||
// This includes emoji, which are essential for social media-style tagging
|
||||
if unicode.IsSymbol(r) {
|
||||
return true
|
||||
}
|
||||
|
||||
// Allow marks (non-spacing, spacing combining, enclosing)
|
||||
// This covers variation selectors (e.g. VS16 \uFE0F) and combining marks (e.g. Keycap \u20E3, accents)
|
||||
if unicode.IsMark(r) {
|
||||
return true
|
||||
}
|
||||
|
||||
// Allow Zero Width Joiner (ZWJ) for emoji sequences
|
||||
if r == '\u200D' {
|
||||
return true
|
||||
}
|
||||
|
||||
// Allow specific ASCII symbols for tag structure
|
||||
// Underscore: word separation (snake_case)
|
||||
// Hyphen: word separation (kebab-case)
|
||||
// Forward slash: hierarchical tags (category/subcategory)
|
||||
// Ampersand: compound tags (science&tech)
|
||||
if r == '_' || r == '-' || r == '/' || r == '&' {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// Parse parses #tag syntax using Unicode-aware validation.
|
||||
// Tags support international characters and follow these rules:
|
||||
// - Must start with # followed by valid tag characters
|
||||
// - Valid characters: Unicode letters, Unicode digits, underscore (_), hyphen (-), forward slash (/)
|
||||
// - Maximum length: 100 runes (Unicode characters)
|
||||
// - Stops at: whitespace, punctuation, or other invalid characters
|
||||
func (*tagParser) Parse(_ gast.Node, block text.Reader, _ parser.Context) gast.Node {
|
||||
line, _ := block.PeekLine()
|
||||
func isXIDContinue(r rune) bool {
|
||||
return containsCodePoint(unicode17XIDContinue[:], r)
|
||||
}
|
||||
|
||||
// Must start with #
|
||||
if len(line) == 0 || line[0] != '#' {
|
||||
return nil
|
||||
}
|
||||
func isDefaultIgnorable(r rune) bool {
|
||||
return containsCodePoint(unicode17DefaultIgnorable[:], r)
|
||||
}
|
||||
|
||||
// Check if it's a heading (## or space after #)
|
||||
if len(line) > 1 {
|
||||
if line[1] == '#' {
|
||||
// It's a heading (##), not a tag
|
||||
return nil
|
||||
}
|
||||
if line[1] == ' ' {
|
||||
// Space after # - heading or just a hash
|
||||
return nil
|
||||
}
|
||||
} else {
|
||||
// Just a lone #
|
||||
return nil
|
||||
}
|
||||
func isCombiningMark(r rune) bool {
|
||||
return containsCodePoint(unicode17CombiningMarks[:], r)
|
||||
}
|
||||
|
||||
// Parse tag using UTF-8 aware rune iteration
|
||||
tagStart := 1
|
||||
pos := tagStart
|
||||
runeCount := 0
|
||||
func isSegmentStarter(r rune) bool {
|
||||
return r == '-' || r == '+' || r == '&' || isXIDContinue(r) && !isCombiningMark(r) && !isDefaultIgnorable(r)
|
||||
}
|
||||
|
||||
for pos < len(line) {
|
||||
r, size := utf8.DecodeRune(line[pos:])
|
||||
func isSegmentContinuation(r rune) bool {
|
||||
return r == '-' || r == '+' || r == '&' || isXIDContinue(r) && !isDefaultIgnorable(r)
|
||||
}
|
||||
|
||||
// Stop at invalid UTF-8
|
||||
func matchFullyQualifiedEmoji(source []byte) int {
|
||||
limit := min(len(source), maxEmoji17SequenceBytes)
|
||||
match := 0
|
||||
for pos := 0; pos < limit; {
|
||||
r, size := utf8.DecodeRune(source[pos:])
|
||||
if r == utf8.RuneError && size == 1 {
|
||||
break
|
||||
}
|
||||
|
||||
// Validate character using Unicode categories
|
||||
if !isValidTagRune(r) {
|
||||
break
|
||||
}
|
||||
|
||||
// Enforce max length (by rune count, not byte count)
|
||||
runeCount++
|
||||
if runeCount > MaxTagLength {
|
||||
break
|
||||
}
|
||||
|
||||
pos += size
|
||||
if pos > limit {
|
||||
break
|
||||
}
|
||||
if _, ok := emoji17FullyQualified[string(source[:pos])]; ok {
|
||||
match = pos
|
||||
}
|
||||
}
|
||||
|
||||
// Must have at least one character after #
|
||||
if pos <= tagStart {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Extract tag (without #)
|
||||
tagName := line[tagStart:pos]
|
||||
|
||||
// Make a copy of the tag name
|
||||
tagCopy := make([]byte, len(tagName))
|
||||
copy(tagCopy, tagName)
|
||||
|
||||
// Advance reader
|
||||
block.Advance(pos)
|
||||
|
||||
// Create node
|
||||
node := &mast.TagNode{
|
||||
Tag: tagCopy,
|
||||
}
|
||||
|
||||
return node
|
||||
return match
|
||||
}
|
||||
|
||||
func scanTagSegment(source []byte) (int, []byte, bool) {
|
||||
pos := 0
|
||||
var value []byte
|
||||
|
||||
starter:
|
||||
for pos < len(source) {
|
||||
if source[pos] == '&' && characterReferenceLength(source[pos:]) > 0 {
|
||||
return 0, nil, false
|
||||
}
|
||||
if emojiLength := matchFullyQualifiedEmoji(source[pos:]); emojiLength > 0 {
|
||||
value = append(value, source[pos:pos+emojiLength]...)
|
||||
pos += emojiLength
|
||||
break
|
||||
}
|
||||
r, size := utf8.DecodeRune(source[pos:])
|
||||
if r == utf8.RuneError && size == 1 {
|
||||
return 0, nil, false
|
||||
}
|
||||
switch {
|
||||
case isDefaultIgnorable(r):
|
||||
pos += size
|
||||
case isXIDContinue(r) && isCombiningMark(r):
|
||||
pos += size
|
||||
case isSegmentStarter(r):
|
||||
value = append(value, source[pos:pos+size]...)
|
||||
pos += size
|
||||
break starter
|
||||
default:
|
||||
return 0, nil, false
|
||||
}
|
||||
}
|
||||
|
||||
if len(value) == 0 {
|
||||
return 0, nil, false
|
||||
}
|
||||
|
||||
for pos < len(source) {
|
||||
if source[pos] == '&' && characterReferenceLength(source[pos:]) > 0 {
|
||||
break
|
||||
}
|
||||
if emojiLength := matchFullyQualifiedEmoji(source[pos:]); emojiLength > 0 {
|
||||
value = append(value, source[pos:pos+emojiLength]...)
|
||||
pos += emojiLength
|
||||
continue
|
||||
}
|
||||
r, size := utf8.DecodeRune(source[pos:])
|
||||
if r == utf8.RuneError && size == 1 {
|
||||
break
|
||||
}
|
||||
switch {
|
||||
case isDefaultIgnorable(r):
|
||||
pos += size
|
||||
case isSegmentContinuation(r):
|
||||
value = append(value, source[pos:pos+size]...)
|
||||
pos += size
|
||||
default:
|
||||
return pos, value, true
|
||||
}
|
||||
}
|
||||
|
||||
return pos, value, true
|
||||
}
|
||||
|
||||
func scanTagIdentifier(source []byte) (int, []byte, bool) {
|
||||
consumed, value, ok := scanTagSegment(source)
|
||||
if !ok {
|
||||
return 0, nil, false
|
||||
}
|
||||
|
||||
for consumed < len(source) && source[consumed] == '/' {
|
||||
segmentLength, segmentValue, segmentOK := scanTagSegment(source[consumed+1:])
|
||||
if !segmentOK {
|
||||
break
|
||||
}
|
||||
value = append(value, '/')
|
||||
value = append(value, segmentValue...)
|
||||
consumed += 1 + segmentLength
|
||||
}
|
||||
|
||||
return consumed, value, true
|
||||
}
|
||||
|
||||
func characterReferenceLength(source []byte) int {
|
||||
if len(source) < 3 || source[0] != '&' {
|
||||
return 0
|
||||
}
|
||||
|
||||
if source[1] == '#' {
|
||||
pos, maxDigits := 2, 7
|
||||
isDigit := func(value byte) bool { return value >= '0' && value <= '9' }
|
||||
if pos < len(source) && (source[pos] == 'x' || source[pos] == 'X') {
|
||||
pos++
|
||||
maxDigits = 6
|
||||
isDigit = func(value byte) bool {
|
||||
return value >= '0' && value <= '9' || value >= 'a' && value <= 'f' || value >= 'A' && value <= 'F'
|
||||
}
|
||||
}
|
||||
start := pos
|
||||
for pos < len(source) && pos-start < maxDigits && isDigit(source[pos]) {
|
||||
pos++
|
||||
}
|
||||
if pos > start && pos < len(source) && source[pos] == ';' {
|
||||
return pos + 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
pos := 1
|
||||
for pos < len(source) && pos <= 32 && util.IsAlphaNumeric(source[pos]) {
|
||||
pos++
|
||||
}
|
||||
if pos == 1 || pos >= len(source) || source[pos] != ';' {
|
||||
return 0
|
||||
}
|
||||
if _, ok := util.LookUpHTML5EntityByName(string(source[1:pos])); !ok {
|
||||
return 0
|
||||
}
|
||||
return pos + 1
|
||||
}
|
||||
|
||||
// FindTagMatches enumerates tag candidates in one eligible
|
||||
// literal-source run. Markdown structure must be resolved before calling it.
|
||||
func FindTagMatches(source []byte) []TagMatch {
|
||||
var matches []TagMatch
|
||||
for pos := 0; pos < len(source); {
|
||||
if emojiLength := matchFullyQualifiedEmoji(source[pos:]); emojiLength > 0 {
|
||||
pos += emojiLength
|
||||
continue
|
||||
}
|
||||
if referenceLength := characterReferenceLength(source[pos:]); referenceLength > 0 {
|
||||
pos += referenceLength
|
||||
continue
|
||||
}
|
||||
if source[pos] == '\\' && pos+1 < len(source) && util.IsPunct(source[pos+1]) {
|
||||
pos += 2
|
||||
continue
|
||||
}
|
||||
if source[pos] != '#' {
|
||||
_, size := utf8.DecodeRune(source[pos:])
|
||||
pos += size
|
||||
continue
|
||||
}
|
||||
|
||||
consumed, value, ok := scanTagIdentifier(source[pos+1:])
|
||||
if !ok {
|
||||
pos++
|
||||
continue
|
||||
}
|
||||
end := pos + consumed + 1
|
||||
matches = append(matches, TagMatch{Start: pos, End: end, Value: value})
|
||||
pos = end
|
||||
}
|
||||
return matches
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,17 +1,16 @@
|
|||
package parser
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/yuin/goldmark/parser"
|
||||
"github.com/yuin/goldmark/text"
|
||||
|
||||
mast "github.com/usememos/memos/internal/markdown/ast"
|
||||
)
|
||||
|
||||
func TestTagParser(t *testing.T) {
|
||||
func TestFindTagMatches(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
|
|
@ -190,57 +189,100 @@ func TestTagParser(t *testing.T) {
|
|||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
p := NewTagParser()
|
||||
reader := text.NewReader([]byte(tt.input))
|
||||
ctx := parser.NewContext()
|
||||
|
||||
node := p.Parse(nil, reader, ctx)
|
||||
|
||||
matches := FindTagMatches([]byte(tt.input))
|
||||
if tt.shouldParse {
|
||||
require.NotNil(t, node, "Expected tag to be parsed")
|
||||
require.IsType(t, &mast.TagNode{}, node)
|
||||
|
||||
tagNode, ok := node.(*mast.TagNode)
|
||||
require.True(t, ok, "Expected node to be *mast.TagNode")
|
||||
assert.Equal(t, tt.expectedTag, string(tagNode.Tag))
|
||||
require.NotEmpty(t, matches, "Expected tag to be parsed")
|
||||
assert.Equal(t, tt.expectedTag, string(matches[0].Value))
|
||||
} else {
|
||||
assert.Nil(t, node, "Expected tag NOT to be parsed")
|
||||
assert.Empty(t, matches, "Expected tag NOT to be parsed")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagParser_Trigger(t *testing.T) {
|
||||
p := NewTagParser()
|
||||
triggers := p.Trigger()
|
||||
|
||||
assert.Equal(t, []byte{'#'}, triggers)
|
||||
func TestFindTagMatchesMultipleTags(t *testing.T) {
|
||||
matches := FindTagMatches([]byte("#tag1 #tag2"))
|
||||
require.Len(t, matches, 2)
|
||||
assert.Equal(t, "tag1", string(matches[0].Value))
|
||||
assert.Equal(t, "tag2", string(matches[1].Value))
|
||||
}
|
||||
|
||||
func TestTagParser_MultipleTags(t *testing.T) {
|
||||
// Test that parser correctly handles multiple tags in sequence
|
||||
input := "#tag1 #tag2"
|
||||
func TestFindTagMatchesMemosTagV1(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
expectedTag string
|
||||
expectedSource string
|
||||
expectedRest string
|
||||
shouldParse bool
|
||||
}{
|
||||
{name: "plus extension", input: "#C++", expectedTag: "C++", expectedSource: "#C++", shouldParse: true},
|
||||
{name: "visible extensions form segment", input: "#---", expectedTag: "---", expectedSource: "#---", shouldParse: true},
|
||||
{name: "ampersands form segment", input: "#&&", expectedTag: "&&", expectedSource: "#&&", shouldParse: true},
|
||||
{name: "Unicode 16 XID character under Unicode 17 profile", input: "#\u1c89", expectedTag: "\u1c89", expectedSource: "#\u1c89", shouldParse: true},
|
||||
{name: "middle dot XID continue", input: "#l·l", expectedTag: "l·l", expectedSource: "#l·l", shouldParse: true},
|
||||
{name: "connector punctuation XID continue", input: "#foo‿bar", expectedTag: "foo‿bar", expectedSource: "#foo‿bar", shouldParse: true},
|
||||
{name: "currency terminates", input: "#price€", expectedTag: "price", expectedSource: "#price", expectedRest: "€", shouldParse: true},
|
||||
{name: "currency cannot start", input: "#€budget", shouldParse: false},
|
||||
{name: "other number terminates", input: "#v²", expectedTag: "v", expectedSource: "#v", expectedRest: "²", shouldParse: true},
|
||||
{name: "leading ZWJ omitted", input: "#\u200dfoo", expectedTag: "foo", expectedSource: "#\u200dfoo", shouldParse: true},
|
||||
{name: "medial ZWJ omitted", input: "#A\u200dB", expectedTag: "AB", expectedSource: "#A\u200dB", shouldParse: true},
|
||||
{name: "medial ZWNJ omitted", input: "#A\u200cB", expectedTag: "AB", expectedSource: "#A\u200cB", shouldParse: true},
|
||||
{name: "variation selector omitted outside emoji", input: "#A\ufe0fB", expectedTag: "AB", expectedSource: "#A\ufe0fB", shouldParse: true},
|
||||
{name: "ignored code points alone", input: "#\u200d\u200c", shouldParse: false},
|
||||
{name: "leading combining mark omitted", input: "#\u0301foo", expectedTag: "foo", expectedSource: "#\u0301foo", shouldParse: true},
|
||||
{name: "leading combining mark alone", input: "#\u0301", shouldParse: false},
|
||||
{name: "combining mark preserved after starter", input: "#cafe\u0301", expectedTag: "cafe\u0301", expectedSource: "#cafe\u0301", shouldParse: true},
|
||||
{name: "ignored prefix restarts after slash", input: "#foo/\u0301bar", expectedTag: "foo/bar", expectedSource: "#foo/\u0301bar", shouldParse: true},
|
||||
{name: "trailing slash", input: "#book/", expectedTag: "book", expectedSource: "#book", expectedRest: "/", shouldParse: true},
|
||||
{name: "leading slash", input: "#/book", shouldParse: false},
|
||||
{name: "repeated slash", input: "#book//fiction", expectedTag: "book", expectedSource: "#book", expectedRest: "//fiction", shouldParse: true},
|
||||
{name: "valid hierarchy before trailing slash", input: "#book/fiction/", expectedTag: "book/fiction", expectedSource: "#book/fiction", expectedRest: "/", shouldParse: true},
|
||||
{name: "fully qualified keycap", input: "#*️⃣", expectedTag: "*️⃣", expectedSource: "#*️⃣", shouldParse: true},
|
||||
{name: "fully qualified emoji", input: "#‼️", expectedTag: "‼️", expectedSource: "#‼️", shouldParse: true},
|
||||
{name: "Emoji 17 sequence", input: "#", expectedTag: "", expectedSource: "#", shouldParse: true},
|
||||
{name: "bare text presentation symbol", input: "#♥", shouldParse: false},
|
||||
{name: "fully qualified heart", input: "#♥️", expectedTag: "♥️", expectedSource: "#♥️", shouldParse: true},
|
||||
{name: "standalone emoji component", input: "#🏻", shouldParse: false},
|
||||
{name: "number sign keycap is not introducer", input: "#️⃣", shouldParse: false},
|
||||
{name: "fullwidth number sign is not introducer", input: "#tag", shouldParse: false},
|
||||
{name: "small number sign is not introducer", input: "﹟tag", shouldParse: false},
|
||||
{name: "keycap as identifier", input: "##️⃣", expectedTag: "#️⃣", expectedSource: "##️⃣", shouldParse: true},
|
||||
{name: "keycap continues identifier", input: "#first#️⃣", expectedTag: "first#️⃣", expectedSource: "#first#️⃣", shouldParse: true},
|
||||
{name: "character reference boundary", input: "#R&D", expectedTag: "R", expectedSource: "#R", expectedRest: "&D", shouldParse: true},
|
||||
{name: "literal ampersand", input: "#R&D", expectedTag: "R&D", expectedSource: "#R&D", shouldParse: true},
|
||||
{
|
||||
name: "no tag length limit",
|
||||
input: "#" + strings.Repeat("a", 101),
|
||||
expectedTag: strings.Repeat("a", 101),
|
||||
expectedSource: "#" + strings.Repeat("a", 101),
|
||||
shouldParse: true,
|
||||
},
|
||||
}
|
||||
|
||||
p := NewTagParser()
|
||||
reader := text.NewReader([]byte(input))
|
||||
ctx := parser.NewContext()
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
matches := FindTagMatches([]byte(tt.input))
|
||||
if !tt.shouldParse {
|
||||
assert.Empty(t, matches)
|
||||
return
|
||||
}
|
||||
|
||||
// Parse first tag
|
||||
node1 := p.Parse(nil, reader, ctx)
|
||||
require.NotNil(t, node1)
|
||||
tagNode1, ok := node1.(*mast.TagNode)
|
||||
require.True(t, ok, "Expected node1 to be *mast.TagNode")
|
||||
assert.Equal(t, "tag1", string(tagNode1.Tag))
|
||||
require.NotEmpty(t, matches)
|
||||
match := matches[0]
|
||||
assert.Equal(t, tt.expectedTag, string(match.Value))
|
||||
assert.Equal(t, tt.expectedSource, tt.input[match.Start:match.End])
|
||||
assert.Equal(t, tt.expectedRest, tt.input[match.End:])
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Advance past the space
|
||||
reader.Advance(1)
|
||||
|
||||
// Parse second tag
|
||||
node2 := p.Parse(nil, reader, ctx)
|
||||
require.NotNil(t, node2)
|
||||
tagNode2, ok := node2.(*mast.TagNode)
|
||||
require.True(t, ok, "Expected node2 to be *mast.TagNode")
|
||||
assert.Equal(t, "tag2", string(tagNode2.Tag))
|
||||
func TestFindTagMatchesRejectsNumericCharacterReference(t *testing.T) {
|
||||
for _, input := range []string{"#tag", "#tag"} {
|
||||
t.Run(input, func(t *testing.T) {
|
||||
assert.Empty(t, FindTagMatches([]byte(input)))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagNode_Kind(t *testing.T) {
|
||||
|
|
|
|||
5121
internal/markdown/parser/tag_unicode_tables.go
Normal file
5121
internal/markdown/parser/tag_unicode_tables.go
Normal file
File diff suppressed because it is too large
Load diff
266
internal/markdown/parser/tagdata/main.go
Normal file
266
internal/markdown/parser/tagdata/main.go
Normal file
|
|
@ -0,0 +1,266 @@
|
|||
// Command tagdata generates the fixed Unicode and emoji tables used by the
|
||||
// Memos tag lexer.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"cmp"
|
||||
"crypto/sha256"
|
||||
"flag"
|
||||
"fmt"
|
||||
"go/format"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
)
|
||||
|
||||
const (
|
||||
unicodeRelease = "17.0.0"
|
||||
derivedCorePropertiesURL = "https://www.unicode.org/Public/" + unicodeRelease + "/ucd/DerivedCoreProperties.txt"
|
||||
unicodeDataURL = "https://www.unicode.org/Public/" + unicodeRelease + "/ucd/UnicodeData.txt"
|
||||
emojiTestURL = "https://www.unicode.org/Public/" + unicodeRelease + "/emoji/emoji-test.txt"
|
||||
)
|
||||
|
||||
type codePointRange struct {
|
||||
lo rune
|
||||
hi rune
|
||||
}
|
||||
|
||||
func main() {
|
||||
output := flag.String("output", "tag_unicode_tables.go", "generated Go output path")
|
||||
flag.Parse()
|
||||
|
||||
client := &http.Client{Timeout: 30 * time.Second}
|
||||
derivedCoreProperties, err := fetch(client, derivedCorePropertiesURL)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
unicodeData, err := fetch(client, unicodeDataURL)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
emojiTest, err := fetch(client, emojiTestURL)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
xidContinue := parseDerivedProperty(derivedCoreProperties, "XID_Continue")
|
||||
defaultIgnorable := parseDerivedProperty(derivedCoreProperties, "Default_Ignorable_Code_Point")
|
||||
combiningMarks := parseCombiningMarks(unicodeData)
|
||||
emojis, maxEmojiBytes := parseFullyQualifiedEmoji(emojiTest)
|
||||
|
||||
var generated bytes.Buffer
|
||||
generated.WriteString("// Code generated by go generate; DO NOT EDIT.\n")
|
||||
fmt.Fprintf(&generated, "// Unicode and Emoji %s data for Memos tag recognition.\n", unicodeRelease)
|
||||
fmt.Fprintf(&generated, "// Unicode data SHA-256: %s\n\n", unicodeDataDigest(xidContinue, defaultIgnorable, combiningMarks, emojis))
|
||||
generated.WriteString("// Sources:\n")
|
||||
fmt.Fprintf(&generated, "// - %s\n", derivedCorePropertiesURL)
|
||||
fmt.Fprintf(&generated, "// - %s\n", unicodeDataURL)
|
||||
fmt.Fprintf(&generated, "// - %s\n\n", emojiTestURL)
|
||||
generated.WriteString("package parser\n\n")
|
||||
writeRanges(&generated, "unicode17XIDContinue", xidContinue)
|
||||
writeRanges(&generated, "unicode17DefaultIgnorable", defaultIgnorable)
|
||||
writeRanges(&generated, "unicode17CombiningMarks", combiningMarks)
|
||||
fmt.Fprintf(&generated, "const maxEmoji17SequenceBytes = %d\n\n", maxEmojiBytes)
|
||||
generated.WriteString("var emoji17FullyQualified = map[string]struct{}{\n")
|
||||
for _, emoji := range emojis {
|
||||
fmt.Fprintf(&generated, "\t%s: {},\n", strconv.QuoteToASCII(emoji))
|
||||
}
|
||||
generated.WriteString("}\n")
|
||||
|
||||
formatted, err := format.Source(generated.Bytes())
|
||||
if err != nil {
|
||||
log.Fatalf("format generated source: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(*output, formatted, 0o644); err != nil {
|
||||
log.Fatalf("write %s: %v", *output, err)
|
||||
}
|
||||
}
|
||||
|
||||
func fetch(client *http.Client, url string) (data []byte, err error) {
|
||||
response, err := client.Get(url)
|
||||
if err != nil {
|
||||
return nil, errors.Wrapf(err, "fetch %s", url)
|
||||
}
|
||||
defer func() {
|
||||
if closeErr := response.Body.Close(); err == nil && closeErr != nil {
|
||||
err = errors.Wrapf(closeErr, "close %s", url)
|
||||
}
|
||||
}()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return nil, errors.Errorf("fetch %s: %s", url, response.Status)
|
||||
}
|
||||
data, err = io.ReadAll(response.Body)
|
||||
if err != nil {
|
||||
return nil, errors.Wrapf(err, "read %s", url)
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
func parseDerivedProperty(data []byte, property string) []codePointRange {
|
||||
var ranges []codePointRange
|
||||
scanner := bufio.NewScanner(bytes.NewReader(data))
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(strings.SplitN(scanner.Text(), "#", 2)[0])
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
fields := strings.Split(line, ";")
|
||||
if len(fields) != 2 || strings.TrimSpace(fields[1]) != property {
|
||||
continue
|
||||
}
|
||||
ranges = append(ranges, parseRange(strings.TrimSpace(fields[0])))
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
log.Fatalf("scan DerivedCoreProperties.txt: %v", err) //nolint:revive // Malformed generator input is fatal.
|
||||
}
|
||||
return mergeRanges(ranges)
|
||||
}
|
||||
|
||||
func parseCombiningMarks(data []byte) []codePointRange {
|
||||
var ranges []codePointRange
|
||||
var rangeStart rune = -1
|
||||
scanner := bufio.NewScanner(bytes.NewReader(data))
|
||||
for scanner.Scan() {
|
||||
if strings.TrimSpace(scanner.Text()) == "" {
|
||||
continue
|
||||
}
|
||||
fields := strings.Split(scanner.Text(), ";")
|
||||
if len(fields) < 3 {
|
||||
log.Fatalf("invalid UnicodeData.txt line: %q", scanner.Text()) //nolint:revive // Malformed generator input is fatal.
|
||||
}
|
||||
codePoint := parseCodePoint(fields[0])
|
||||
name := fields[1]
|
||||
category := fields[2]
|
||||
isCombining := category == "Mn" || category == "Mc"
|
||||
if strings.HasSuffix(name, ", First>") {
|
||||
if isCombining {
|
||||
rangeStart = codePoint
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.HasSuffix(name, ", Last>") {
|
||||
if rangeStart >= 0 {
|
||||
ranges = append(ranges, codePointRange{lo: rangeStart, hi: codePoint})
|
||||
rangeStart = -1
|
||||
}
|
||||
continue
|
||||
}
|
||||
if isCombining {
|
||||
ranges = append(ranges, codePointRange{lo: codePoint, hi: codePoint})
|
||||
}
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
log.Fatalf("scan UnicodeData.txt: %v", err) //nolint:revive // Malformed generator input is fatal.
|
||||
}
|
||||
return mergeRanges(ranges)
|
||||
}
|
||||
|
||||
func unicodeDataDigest(xidContinue, defaultIgnorable, combiningMarks []codePointRange, emojis []string) string {
|
||||
var canonical strings.Builder
|
||||
writeRanges := func(name string, ranges []codePointRange) {
|
||||
for _, codeRange := range ranges {
|
||||
fmt.Fprintf(&canonical, "%s:%x-%x\n", name, codeRange.lo, codeRange.hi)
|
||||
}
|
||||
}
|
||||
writeRanges("xid", xidContinue)
|
||||
writeRanges("ignorable", defaultIgnorable)
|
||||
writeRanges("combining", combiningMarks)
|
||||
|
||||
emojiLines := make([]string, 0, len(emojis))
|
||||
for _, emoji := range emojis {
|
||||
codePoints := make([]string, 0, len(emoji))
|
||||
for _, codePoint := range emoji {
|
||||
codePoints = append(codePoints, strconv.FormatInt(int64(codePoint), 16))
|
||||
}
|
||||
emojiLines = append(emojiLines, "emoji:"+strings.Join(codePoints, ",")+"\n")
|
||||
}
|
||||
slices.Sort(emojiLines)
|
||||
for _, line := range emojiLines {
|
||||
canonical.WriteString(line)
|
||||
}
|
||||
|
||||
digest := sha256.Sum256([]byte(canonical.String()))
|
||||
return fmt.Sprintf("%x", digest)
|
||||
}
|
||||
|
||||
func parseFullyQualifiedEmoji(data []byte) ([]string, int) {
|
||||
var emojis []string
|
||||
maxBytes := 0
|
||||
scanner := bufio.NewScanner(bytes.NewReader(data))
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(strings.SplitN(scanner.Text(), "#", 2)[0])
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
fields := strings.Split(line, ";")
|
||||
if len(fields) != 2 || strings.TrimSpace(fields[1]) != "fully-qualified" {
|
||||
continue
|
||||
}
|
||||
var sequence strings.Builder
|
||||
for _, field := range strings.Fields(fields[0]) {
|
||||
sequence.WriteRune(parseCodePoint(field))
|
||||
}
|
||||
emoji := sequence.String()
|
||||
emojis = append(emojis, emoji)
|
||||
maxBytes = max(maxBytes, len(emoji))
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
log.Fatalf("scan emoji-test.txt: %v", err) //nolint:revive // Malformed generator input is fatal.
|
||||
}
|
||||
slices.Sort(emojis)
|
||||
return emojis, maxBytes
|
||||
}
|
||||
|
||||
func parseRange(value string) codePointRange {
|
||||
parts := strings.Split(value, "..")
|
||||
if len(parts) == 1 {
|
||||
codePoint := parseCodePoint(parts[0])
|
||||
return codePointRange{lo: codePoint, hi: codePoint}
|
||||
}
|
||||
if len(parts) != 2 {
|
||||
log.Fatalf("invalid code-point range %q", value) //nolint:revive // Malformed generator input is fatal.
|
||||
}
|
||||
return codePointRange{lo: parseCodePoint(parts[0]), hi: parseCodePoint(parts[1])}
|
||||
}
|
||||
|
||||
func parseCodePoint(value string) rune {
|
||||
codePoint, err := strconv.ParseInt(strings.TrimSpace(value), 16, 32)
|
||||
if err != nil {
|
||||
log.Fatalf("parse code point %q: %v", value, err) //nolint:revive // Malformed generator input is fatal.
|
||||
}
|
||||
return rune(codePoint)
|
||||
}
|
||||
|
||||
func mergeRanges(ranges []codePointRange) []codePointRange {
|
||||
if len(ranges) == 0 {
|
||||
return nil
|
||||
}
|
||||
slices.SortFunc(ranges, func(a, b codePointRange) int { return cmp.Compare(a.lo, b.lo) })
|
||||
merged := []codePointRange{ranges[0]}
|
||||
for _, current := range ranges[1:] {
|
||||
last := &merged[len(merged)-1]
|
||||
if current.lo <= last.hi+1 {
|
||||
last.hi = max(last.hi, current.hi)
|
||||
continue
|
||||
}
|
||||
merged = append(merged, current)
|
||||
}
|
||||
return merged
|
||||
}
|
||||
|
||||
func writeRanges(output *bytes.Buffer, name string, ranges []codePointRange) {
|
||||
fmt.Fprintf(output, "var %s = [...]codePointRange{\n", name)
|
||||
for _, codeRange := range ranges {
|
||||
fmt.Fprintf(output, "\t{lo: 0x%04X, hi: 0x%04X},\n", codeRange.lo, codeRange.hi)
|
||||
}
|
||||
output.WriteString("}\n\n")
|
||||
}
|
||||
31
internal/markdown/parser/tagdata/main_test.go
Normal file
31
internal/markdown/parser/tagdata/main_test.go
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestParseDerivedPropertyMergesMatchingRanges(t *testing.T) {
|
||||
data := []byte("0041..0042 ; XID_Continue # letters\n0043 ; XID_Continue\n200D ; Default_Ignorable_Code_Point\n")
|
||||
|
||||
require.Equal(t, []codePointRange{{lo: 0x41, hi: 0x43}}, parseDerivedProperty(data, "XID_Continue"))
|
||||
}
|
||||
|
||||
func TestParseCombiningMarksIncludesSinglesAndRanges(t *testing.T) {
|
||||
data := []byte("\n0300;COMBINING GRAVE ACCENT;Mn\n0903;DEVANAGARI SIGN VISARGA;Mc\n1000;<TEST, First>;Mn\n1002;<TEST, Last>;Mn\n0041;LATIN CAPITAL LETTER A;Lu\n\n")
|
||||
|
||||
require.Equal(t, []codePointRange{
|
||||
{lo: 0x0300, hi: 0x0300},
|
||||
{lo: 0x0903, hi: 0x0903},
|
||||
{lo: 0x1000, hi: 0x1002},
|
||||
}, parseCombiningMarks(data))
|
||||
}
|
||||
|
||||
func TestParseFullyQualifiedEmojiIgnoresOtherStatuses(t *testing.T) {
|
||||
data := []byte("1F600 ; fully-qualified # grinning face\n263A FE0F ; fully-qualified # smiling face\n1F3F4 E0067 E0062 E0073 E0063 E0074 E007F ; fully-qualified # flag: Scotland\n263A ; unqualified\n1F3FB ; component\n")
|
||||
|
||||
emojis, maxBytes := parseFullyQualifiedEmoji(data)
|
||||
require.ElementsMatch(t, []string{"😀", "☺️", "🏴"}, emojis)
|
||||
require.Equal(t, len("🏴"), maxBytes)
|
||||
}
|
||||
|
|
@ -39,7 +39,15 @@ func (r *MarkdownRenderer) renderNode(node gast.Node, source []byte, depth int)
|
|||
case *gast.Paragraph:
|
||||
r.renderChildren(n, source, depth)
|
||||
if node.NextSibling() != nil {
|
||||
r.buf.WriteString("\n\n")
|
||||
separator := "\n\n"
|
||||
if _, nextIsList := node.NextSibling().(*gast.List); nextIsList {
|
||||
if item, ok := node.Parent().(*gast.ListItem); ok {
|
||||
if list, ok := item.Parent().(*gast.List); ok && list.IsTight {
|
||||
separator = "\n"
|
||||
}
|
||||
}
|
||||
}
|
||||
r.buf.WriteString(separator)
|
||||
}
|
||||
|
||||
case *gast.Text:
|
||||
|
|
@ -79,11 +87,19 @@ func (r *MarkdownRenderer) renderNode(node gast.Node, source []byte, depth int)
|
|||
r.buf.WriteString(")")
|
||||
|
||||
case *gast.AutoLink:
|
||||
if start, end, ok := autoLinkLabelRange(n, source); ok {
|
||||
if start > 0 && end < len(source) && source[start-1] == '<' && source[end] == '>' {
|
||||
r.buf.Write(source[start-1 : end+1])
|
||||
} else {
|
||||
r.buf.Write(source[start:end])
|
||||
}
|
||||
break
|
||||
}
|
||||
url := n.URL(source)
|
||||
if n.AutoLinkType == gast.AutoLinkEmail {
|
||||
r.buf.WriteString("<")
|
||||
r.buf.WriteByte('<')
|
||||
r.buf.Write(url)
|
||||
r.buf.WriteString(">")
|
||||
r.buf.WriteByte('>')
|
||||
} else {
|
||||
r.buf.Write(url)
|
||||
}
|
||||
|
|
@ -119,6 +135,9 @@ func (r *MarkdownRenderer) renderNode(node gast.Node, source []byte, depth int)
|
|||
}
|
||||
|
||||
case *gast.List:
|
||||
if node.PreviousSibling() != nil && !bytes.HasSuffix(r.buf.Bytes(), []byte{'\n'}) {
|
||||
r.buf.WriteByte('\n')
|
||||
}
|
||||
r.renderChildren(n, source, depth)
|
||||
if node.NextSibling() != nil {
|
||||
r.buf.WriteString("\n\n")
|
||||
|
|
@ -153,8 +172,27 @@ func (r *MarkdownRenderer) renderNode(node gast.Node, source []byte, depth int)
|
|||
|
||||
// Custom Memos nodes
|
||||
case *mast.TagNode:
|
||||
r.buf.WriteByte('#')
|
||||
r.buf.Write(n.Tag)
|
||||
if len(n.Source) > 0 {
|
||||
r.buf.Write(n.Source)
|
||||
} else {
|
||||
r.buf.WriteByte('#')
|
||||
r.buf.Write(n.Tag)
|
||||
}
|
||||
|
||||
case *mast.GFMEmailNode:
|
||||
r.buf.Write(n.Source)
|
||||
|
||||
case *mast.InlineMathNode:
|
||||
r.buf.Write(n.Source)
|
||||
|
||||
case *mast.BlockMathNode:
|
||||
r.buf.Write(n.Source)
|
||||
if node.NextSibling() != nil {
|
||||
if !bytes.HasSuffix(n.Source, []byte{'\n'}) {
|
||||
r.buf.WriteByte('\n')
|
||||
}
|
||||
r.buf.WriteByte('\n')
|
||||
}
|
||||
|
||||
case *mast.MentionNode:
|
||||
r.buf.WriteByte('@')
|
||||
|
|
@ -166,6 +204,16 @@ func (r *MarkdownRenderer) renderNode(node gast.Node, source []byte, depth int)
|
|||
}
|
||||
}
|
||||
|
||||
func autoLinkLabelRange(node *gast.AutoLink, source []byte) (int, int, bool) {
|
||||
label := node.Label(source)
|
||||
for _, start := range []int{node.Pos(), node.Pos() + 1} {
|
||||
if start >= 0 && start <= len(source)-len(label) && bytes.HasPrefix(source[start:], label) {
|
||||
return start, start + len(label), true
|
||||
}
|
||||
}
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
// renderChildren renders all children of a node.
|
||||
func (r *MarkdownRenderer) renderChildren(node gast.Node, source []byte, depth int) {
|
||||
child := node.FirstChild()
|
||||
|
|
@ -238,26 +286,41 @@ func (r *MarkdownRenderer) renderListItem(node *gast.ListItem, source []byte, de
|
|||
return
|
||||
}
|
||||
|
||||
// Add indentation only for nested lists
|
||||
// Document=0, List=1, ListItem=2 (no indent), nested ListItem=3+ (indent)
|
||||
// Document=0, List=1, ListItem=2 (no indent), nested ListItem=3+ (indent).
|
||||
prefix := ""
|
||||
if depth > 2 {
|
||||
indent := strings.Repeat(" ", depth-2)
|
||||
r.buf.WriteString(indent)
|
||||
prefix = strings.Repeat(" ", (depth-2)/2)
|
||||
}
|
||||
|
||||
// Add list marker
|
||||
marker := "- "
|
||||
if list.IsOrdered() {
|
||||
fmt.Fprintf(r.buf, "%d. ", list.Start)
|
||||
marker = fmt.Sprintf("%d. ", list.Start)
|
||||
list.Start++ // Increment for next item
|
||||
} else {
|
||||
r.buf.WriteString("- ")
|
||||
}
|
||||
|
||||
// Render content
|
||||
r.renderChildren(node, source, depth)
|
||||
contentBuffer := &bytes.Buffer{}
|
||||
contentRenderer := &MarkdownRenderer{buf: contentBuffer}
|
||||
contentRenderer.renderChildren(node, source, depth)
|
||||
content := contentBuffer.String()
|
||||
contentLines := strings.Split(content, "\n")
|
||||
|
||||
r.buf.WriteString(prefix)
|
||||
r.buf.WriteString(marker)
|
||||
r.buf.WriteString(contentLines[0])
|
||||
continuationIndent := strings.Repeat(" ", len(prefix)+len(marker))
|
||||
for index, line := range contentLines[1:] {
|
||||
r.buf.WriteByte('\n')
|
||||
if line != "" || index < len(contentLines)-2 {
|
||||
lineIndent := len(line) - len(strings.TrimLeft(line, " "))
|
||||
if lineIndent < len(continuationIndent) {
|
||||
r.buf.WriteString(continuationIndent[:len(continuationIndent)-lineIndent])
|
||||
}
|
||||
r.buf.WriteString(line)
|
||||
}
|
||||
}
|
||||
|
||||
// Add newline if there's a next sibling
|
||||
if node.NextSibling() != nil {
|
||||
if node.NextSibling() != nil && !strings.HasSuffix(content, "\n") {
|
||||
r.buf.WriteByte('\n')
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -80,6 +80,16 @@ func TestMarkdownRenderer(t *testing.T) {
|
|||
input: "1. First\n2. Second\n3. Third",
|
||||
expected: "1. First\n2. Second\n3. Third",
|
||||
},
|
||||
{
|
||||
name: "nested unordered list",
|
||||
input: "- Parent\n - Child\n - Second child\n- Sibling",
|
||||
expected: "- Parent\n - Child\n - Second child\n- Sibling",
|
||||
},
|
||||
{
|
||||
name: "nested ordered list",
|
||||
input: "1. Parent\n 1. Child\n2. Sibling",
|
||||
expected: "1. Parent\n 1. Child\n2. Sibling",
|
||||
},
|
||||
{
|
||||
name: "blockquote",
|
||||
input: "> This is a quote\n> Second line",
|
||||
|
|
|
|||
|
|
@ -38,6 +38,39 @@ func TestCreateMemoAcceptsUUID(t *testing.T) {
|
|||
require.Equal(t, "memos/"+memoID, memo.Name)
|
||||
}
|
||||
|
||||
func TestCreateAndUpdateMemoRebuildsTagPayload(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ts := NewTestService(t)
|
||||
defer ts.Cleanup()
|
||||
|
||||
user, err := ts.CreateRegularUser(ctx, "tag-payload-user")
|
||||
require.NoError(t, err)
|
||||
userCtx := ts.CreateUserContext(ctx, user.ID)
|
||||
|
||||
memo, err := ts.Service.CreateMemo(userCtx, &apiv1.CreateMemoRequest{
|
||||
Memo: &apiv1.Memo{
|
||||
Content: "#book/fiction #Work #work #A\u200dB https://example.com/#hidden",
|
||||
Visibility: apiv1.Visibility_PRIVATE,
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, []string{"book", "book/fiction", "Work", "work", "AB"}, memo.Tags)
|
||||
|
||||
memo, err = ts.Service.UpdateMemo(userCtx, &apiv1.UpdateMemoRequest{
|
||||
Memo: &apiv1.Memo{
|
||||
Name: memo.Name,
|
||||
Content: "#next #A\u200dB",
|
||||
},
|
||||
UpdateMask: &fieldmaskpb.FieldMask{Paths: []string{"content"}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, []string{"next", "AB"}, memo.Tags)
|
||||
|
||||
stored, err := ts.Service.GetMemo(userCtx, &apiv1.GetMemoRequest{Name: memo.Name})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, []string{"next", "AB"}, stored.Tags)
|
||||
}
|
||||
|
||||
func TestListMemos(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ func TestGetUserStats_TagCount(t *testing.T) {
|
|||
Content: "This is a test memo with #test tag",
|
||||
Visibility: store.Public,
|
||||
Payload: &storepb.MemoPayload{
|
||||
Tags: []string{"test"},
|
||||
Tags: []string{"test", "test"},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
|
@ -48,7 +48,8 @@ func TestGetUserStats_TagCount(t *testing.T) {
|
|||
require.NotNil(t, response)
|
||||
require.Equal(t, fmt.Sprintf("users/%s/stats", user.Username), response.Name)
|
||||
|
||||
// Check that the tag count is exactly 1, not 2
|
||||
// A memo contributes at most once to an exact tag count, even if its payload
|
||||
// accidentally contains the same derived membership more than once.
|
||||
require.Contains(t, response.TagCount, "test")
|
||||
require.Equal(t, int32(1), response.TagCount["test"], "Tag count should be 1 for a single occurrence")
|
||||
|
||||
|
|
@ -111,6 +112,44 @@ func TestGetUserStats_TagCount(t *testing.T) {
|
|||
require.Contains(t, err.Error(), "user not found")
|
||||
}
|
||||
|
||||
func TestGetUserStats_TagCountPreservesExactIdentity(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ts := NewTestService(t)
|
||||
defer ts.Cleanup()
|
||||
const (
|
||||
composedTag = "caf\u00e9"
|
||||
decomposedTag = "cafe\u0301"
|
||||
)
|
||||
|
||||
user, err := ts.CreateHostUser(ctx, "exact-tag-stats-user")
|
||||
require.NoError(t, err)
|
||||
userCtx := ts.CreateUserContext(ctx, user.ID)
|
||||
|
||||
_, err = ts.Store.CreateMemo(ctx, &store.Memo{
|
||||
UID: "exact-tag-stats-memo",
|
||||
CreatorID: user.ID,
|
||||
Content: "Exact tag membership fixture",
|
||||
Visibility: store.Public,
|
||||
Payload: &storepb.MemoPayload{
|
||||
Tags: []string{"book", "book/fiction", "book", "Work", "work", composedTag, decomposedTag},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
response, err := ts.Service.GetUserStats(userCtx, &v1pb.GetUserStatsRequest{
|
||||
Name: fmt.Sprintf("users/%s", user.Username),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, map[string]int32{
|
||||
"book": 1,
|
||||
"book/fiction": 1,
|
||||
"Work": 1,
|
||||
"work": 1,
|
||||
composedTag: 1,
|
||||
decomposedTag: 1,
|
||||
}, response.TagCount)
|
||||
}
|
||||
|
||||
func TestGetUserStats_MemoUpdatedTimestamps(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
|
|
@ -197,7 +236,7 @@ func TestListAllUserStats_FilterExcludesPrivateMemos(t *testing.T) {
|
|||
CreatorID: user.ID,
|
||||
Content: "public memo",
|
||||
Visibility: store.Public,
|
||||
Payload: &storepb.MemoPayload{Tags: []string{"public"}},
|
||||
Payload: &storepb.MemoPayload{Tags: []string{"public", "public"}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
_, err = ts.Store.CreateMemo(ctx, &store.Memo{
|
||||
|
|
@ -205,7 +244,7 @@ func TestListAllUserStats_FilterExcludesPrivateMemos(t *testing.T) {
|
|||
CreatorID: user.ID,
|
||||
Content: "private memo",
|
||||
Visibility: store.Private,
|
||||
Payload: &storepb.MemoPayload{Tags: []string{"private"}},
|
||||
Payload: &storepb.MemoPayload{Tags: []string{"private", "private"}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
|
|
|
|||
|
|
@ -135,9 +135,7 @@ func (s *APIV1Service) ListAllUserStats(ctx context.Context, request *v1pb.ListA
|
|||
|
||||
// Count tags and other properties
|
||||
if memo.Payload != nil {
|
||||
for _, tag := range memo.Payload.Tags {
|
||||
stats.TagCount[tag]++
|
||||
}
|
||||
incrementTagCounts(stats.TagCount, memo.Payload.Tags)
|
||||
if memo.Payload.Property != nil {
|
||||
if memo.Payload.Property.HasLink {
|
||||
stats.MemoTypeStats.LinkCount++
|
||||
|
|
@ -251,9 +249,7 @@ func (s *APIV1Service) GetUserStats(ctx context.Context, request *v1pb.GetUserSt
|
|||
updatedTimestamps = append(updatedTimestamps, timestamppb.New(time.Unix(memo.UpdatedTs, 0)))
|
||||
// Count different memo types based on content.
|
||||
if memo.Payload != nil {
|
||||
for _, tag := range memo.Payload.Tags {
|
||||
tagCount[tag]++
|
||||
}
|
||||
incrementTagCounts(tagCount, memo.Payload.Tags)
|
||||
if memo.Payload.Property != nil {
|
||||
if memo.Payload.Property.HasLink {
|
||||
linkCount++
|
||||
|
|
@ -294,3 +290,15 @@ func (s *APIV1Service) GetUserStats(ctx context.Context, request *v1pb.GetUserSt
|
|||
|
||||
return userStats, nil
|
||||
}
|
||||
|
||||
// incrementTagCounts counts each distinct tag at most once for one memo payload.
|
||||
func incrementTagCounts(counts map[string]int32, tags []string) {
|
||||
seen := make(map[string]struct{}, len(tags))
|
||||
for _, tag := range tags {
|
||||
if _, ok := seen[tag]; ok {
|
||||
continue
|
||||
}
|
||||
seen[tag] = struct{}{}
|
||||
counts[tag]++
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -37,13 +37,13 @@ INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,pin
|
|||
-- ============================================================================
|
||||
|
||||
-- 10. Steven: official Chrome extension launch (featured first in Explore)
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(10,'webclipper0001',1,strftime('%s','now','-30 minutes'),strftime('%s','now','-30 minutes'),replace('# ✂️ Memos Web Clipper for Chrome\n\nThe ***official*** Memos Web Clipper turns the page you''re reading into a source-linked memo in under five seconds.\n\n[Add Memos Web Clipper to Chrome →](https://chromewebstore.google.com/detail/memos-web-clipper/nebaoebnljalfegiidibihhkebeiklbl)\n\n| Capture from Chrome | What lands in Memos |\n|---------------------|---------------------|\n| Current page | Title, summary, and source link |\n| Text selection | Quoted excerpt with its source |\n| Selected image | A real attachment on your instance |\n\nEdit the generated memo, choose its visibility, or customize the template and default tags before saving.\n\n### Private by design\n\nThe clipper sends captured content only to the self-hosted Memos instance you connect — and only when you save. No analytics, third-party servers, or tracking. #memos/extensions','\n',char(10)),'PUBLIC','{"tags":["memos/extensions"],"property":{"hasLink":true}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(10,'webclipper0001',1,strftime('%s','now','-30 minutes'),strftime('%s','now','-30 minutes'),replace('# ✂️ Memos Web Clipper for Chrome\n\nThe ***official*** Memos Web Clipper turns the page you''re reading into a source-linked memo in under five seconds.\n\n[Add Memos Web Clipper to Chrome →](https://chromewebstore.google.com/detail/memos-web-clipper/nebaoebnljalfegiidibihhkebeiklbl)\n\n| Capture from Chrome | What lands in Memos |\n|---------------------|---------------------|\n| Current page | Title, summary, and source link |\n| Text selection | Quoted excerpt with its source |\n| Selected image | A real attachment on your instance |\n\nEdit the generated memo, choose its visibility, or customize the template and default tags before saving.\n\n### Private by design\n\nThe clipper sends captured content only to the self-hosted Memos instance you connect — and only when you save. No analytics, third-party servers, or tracking. #memos/extensions','\n',char(10)),'PUBLIC','{"tags":["memos","memos/extensions"],"property":{"hasLink":true}}');
|
||||
|
||||
-- 3. Ben: photo memo with an image attachment + location
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(3,'goldenhour0001',3,strftime('%s','now','-3 hours'),strftime('%s','now','-3 hours'),'Golden hour on the coastal trail tonight. The camera couldn''t hold the colors, so here''s my minimal-art rendition instead 🎨 #travel/hikes','PUBLIC','{"tags":["travel/hikes"],"property":{"hasLink":false},"location":{"placeholder":"Point Reyes, California, United States","latitude":37.9954,"longitude":-123.0204}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(3,'goldenhour0001',3,strftime('%s','now','-3 hours'),strftime('%s','now','-3 hours'),'Golden hour on the coastal trail tonight. The camera couldn''t hold the colors, so here''s my minimal-art rendition instead 🎨 #travel/hikes','PUBLIC','{"tags":["travel","travel/hikes"],"property":{"hasLink":false},"location":{"placeholder":"Point Reyes, California, United States","latitude":37.9954,"longitude":-123.0204}}');
|
||||
|
||||
-- 27. Steven: build-in-public changelog note (real v0.30 highlights)
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(27,'v030changelog1',1,strftime('%s','now','-6 hours'),strftime('%s','now','-6 hours'),replace('Rolled v0.30 out to this demo instance 🚀 The highlights:\n\n- Editor rebuilt on CodeMirror — your markdown stays plain text, but formatting renders as you type\n- Scroll position is restored when you navigate back to a long feed\n- Tooltip polish and a pile of small fixes\n\nFull notes: [github.com/usememos/memos/releases](https://github.com/usememos/memos/releases)\n\nDogfooding note: this memo was of course drafted in the new editor. #memos/changelog','\n',char(10)),'PUBLIC','{"tags":["memos/changelog"],"property":{"hasLink":true}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(27,'v030changelog1',1,strftime('%s','now','-6 hours'),strftime('%s','now','-6 hours'),replace('Rolled v0.30 out to this demo instance 🚀 The highlights:\n\n- Editor rebuilt on CodeMirror — your markdown stays plain text, but formatting renders as you type\n- Scroll position is restored when you navigate back to a long feed\n- Tooltip polish and a pile of small fixes\n\nFull notes: [github.com/usememos/memos/releases](https://github.com/usememos/memos/releases)\n\nDogfooding note: this memo was of course drafted in the new editor. #memos/changelog','\n',char(10)),'PUBLIC','{"tags":["memos","memos/changelog"],"property":{"hasLink":true}}');
|
||||
|
||||
-- 4. Zoe: one-line book quote
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(4,'zoequote000001',4,strftime('%s','now','-7 hours'),strftime('%s','now','-7 hours'),replace('> "We tell ourselves stories in order to live." — Joan Didion\n\nBeen thinking about this all morning. #books','\n',char(10)),'PUBLIC','{"tags":["books"],"property":{"hasLink":false}}');
|
||||
|
|
@ -55,19 +55,19 @@ INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,pay
|
|||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(6,'tagtipmemo0001',1,strftime('%s','now','-26 hours'),strftime('%s','now','-26 hours'),'Tip: tags are just typed inline — `#like/this` — and nested ones grow a tree in the sidebar. No setup, no tag manager. #getting-started','PUBLIC','{"tags":["getting-started"],"property":{"hasLink":false}}');
|
||||
|
||||
-- 7. Alice: shipped-it note
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(7,'aliceshipped01',2,strftime('%s','now','-2 days','+3 hours'),strftime('%s','now','-2 days','+3 hours'),'Shipped v0.2 of my side project tonight 🚀 Rewrote the sync engine and cut startup time in half. Scrolling back through my own #dev/til memos made the changelog write itself.','PUBLIC','{"tags":["dev/til"],"property":{"hasLink":false}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(7,'aliceshipped01',2,strftime('%s','now','-2 days','+3 hours'),strftime('%s','now','-2 days','+3 hours'),'Shipped v0.2 of my side project tonight 🚀 Rewrote the sync engine and cut startup time in half. Scrolling back through my own #dev/til memos made the changelog write itself.','PUBLIC','{"tags":["dev","dev/til"],"property":{"hasLink":false}}');
|
||||
|
||||
-- 8. Alice: short TIL with a code block (referenced by the weekly review)
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(8,'alicetilgit001',2,strftime('%s','now','-3 days'),strftime('%s','now','-3 days'),replace('TIL you can diff a single file between branches — no checkout, no stash dance:\n\n```bash\ngit diff main..feature -- path/to/file.go\n```\n\n#dev/til #dev/git','\n',char(10)),'PUBLIC','{"tags":["dev/til","dev/git"],"property":{"hasLink":false,"hasCode":true}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(8,'alicetilgit001',2,strftime('%s','now','-3 days'),strftime('%s','now','-3 days'),replace('TIL you can diff a single file between branches — no checkout, no stash dance:\n\n```bash\ngit diff main..feature -- path/to/file.go\n```\n\n#dev/til #dev/git','\n',char(10)),'PUBLIC','{"tags":["dev","dev/til","dev/git"],"property":{"hasLink":false,"hasCode":true}}');
|
||||
|
||||
-- 29. Alice: link/bookmark capture — a core Memos use case
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(29,'alicebookmark1',2,strftime('%s','now','-32 hours'),strftime('%s','now','-32 hours'),'Weekend read, bookmarking before I lose it: [SQLite: 35% faster than the filesystem](https://sqlite.org/fasterthanfs.html) — makes me feel better about my entire digital life being one .db file 😄 #dev/reading','PUBLIC','{"tags":["dev/reading"],"property":{"hasLink":true}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(29,'alicebookmark1',2,strftime('%s','now','-32 hours'),strftime('%s','now','-32 hours'),'Weekend read, bookmarking before I lose it: [SQLite: 35% faster than the filesystem](https://sqlite.org/fasterthanfs.html) — makes me feel better about my entire digital life being one .db file 😄 #dev/reading','PUBLIC','{"tags":["dev","dev/reading"],"property":{"hasLink":true}}');
|
||||
|
||||
-- 9. Zoe: recipe with a task list
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(9,'zoeramen000001',4,strftime('%s','now','-4 days'),strftime('%s','now','-4 days'),replace('## 🍜 Midnight miso ramen (15 minutes)\n\nFor nights when the book was too good to stop reading.\n\n- [x] Dashi or veg stock, 500ml\n- [x] Miso paste, 2 big spoons\n- [x] Noodles, one nest\n- [x] Soft egg — 6½ minutes, exactly\n- [ ] Scallions (always forget the scallions)\n\nSteep the miso, don''t boil it. That''s the whole secret. #cooking','\n',char(10)),'PUBLIC','{"tags":["cooking"],"property":{"hasLink":false,"hasTaskList":true,"hasIncompleteTasks":true}}');
|
||||
|
||||
-- 30. Steven: compact Markdown rendering check framed as maintainer dogfooding
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(30,'rendertest0001',1,strftime('%s','now','-10 days'),strftime('%s','now','-10 days'),replace('## 🧪 Markdown rendering check #memos/dev\n\nA quick smoke test I open after releases:\n\n**Bold**, *italic*, ~~strikethrough~~, `inline code`, and a [docs link](https://www.usememos.com/docs).\n\n> Simple notes should render beautifully.\n\n```go\nfmt.Println("syntax highlighting ✓")\n```\n\n| Check | Status |\n|-------|--------|\n| Text styles | ✅ |\n| Code blocks | ✅ |\n| Tables | ✅ |\n\n- [x] Check desktop\n- [x] Check mobile\n- [ ] Ship the next release','\n',char(10)),'PUBLIC','{"tags":["memos/dev"],"property":{"hasLink":true,"hasCode":true,"hasTaskList":true,"hasIncompleteTasks":true}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(30,'rendertest0001',1,strftime('%s','now','-10 days'),strftime('%s','now','-10 days'),replace('## 🧪 Markdown rendering check #memos/dev\n\nA quick smoke test I open after releases:\n\n**Bold**, *italic*, ~~strikethrough~~, `inline code`, and a [docs link](https://www.usememos.com/docs).\n\n> Simple notes should render beautifully.\n\n```go\nfmt.Println("syntax highlighting ✓")\n```\n\n| Check | Status |\n|-------|--------|\n| Text styles | ✅ |\n| Code blocks | ✅ |\n| Tables | ✅ |\n\n- [x] Check desktop\n- [x] Check mobile\n- [ ] Ship the next release','\n',char(10)),'PUBLIC','{"tags":["memos","memos/dev"],"property":{"hasLink":true,"hasCode":true,"hasTaskList":true,"hasIncompleteTasks":true}}');
|
||||
|
||||
-- 11. Zoe: movie watchlist with a table
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(11,'zoemovies00001',4,strftime('%s','now','-9 days'),strftime('%s','now','-9 days'),replace('## 🎬 Movie catch-up #movies\n\nWorking through films I''ve been meaning to watch for, uh, years.\n\n| Movie | Status | Rating |\n|-------|--------|--------|\n| The Grand Budapest Hotel | ✅ Watched | ⭐⭐⭐⭐⭐ |\n| Spirited Away | ✅ Watched | ⭐⭐⭐⭐⭐ |\n| Everything Everywhere All at Once | ✅ Watched | ⭐⭐⭐⭐⭐ |\n| Dune: Part Two | 📅 This weekend | — |\n| Oppenheimer | 📋 Queued | — |\n\n**Spirited Away**: cried twice, no regrets. **Everything Everywhere**: went in cold, did not expect the bagel.\n\nNext month: full Miyazaki marathon 🎨 Taking suggestions for where to start.','\n',char(10)),'PUBLIC','{"tags":["movies"],"property":{"hasLink":false}}');
|
||||
|
|
@ -76,10 +76,10 @@ INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,pay
|
|||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(12,'bennighttrain1',3,strftime('%s','now','-12 days'),strftime('%s','now','-12 days'),'Overnight train from Oslo to Bergen. Woke at 5am to fjords in fog and couldn''t go back to sleep. Some views are worth the bad coffee ☕ #travel','PUBLIC','{"tags":["travel"],"property":{"hasLink":false},"location":{"placeholder":"Bergen, Vestland, Norway","latitude":60.3913,"longitude":5.3221}}');
|
||||
|
||||
-- 13. Ben: travel bucket list + location
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(13,'benbucketlist1',3,strftime('%s','now','-13 days'),strftime('%s','now','-13 days'),replace('## 🌍 My travel bucket list #travel/bucketlist\n\n> Writing this from a tiny café near the Seine. Will keep updating after each trip. 📍\n\n### Been\n- [x] Paris, France — the croissants ruined every croissant after\n- [x] Shanghai, China — modern skyline next to thousand-year-old temples, somehow it works\n- [x] Grand Canyon, USA — photos do not do it justice, and I say that professionally\n- [x] Barcelona, Spain — Gaudí''s buildings look grown, not built\n\n### Next\n- [ ] Northern Lights in Iceland — booking for winter, *finally*\n- [ ] Safari in Tanzania\n- [ ] Great Barrier Reef, before it gets worse\n- [ ] New Zealand road trip — south island, two weeks minimum','\n',char(10)),'PUBLIC','{"tags":["travel/bucketlist"],"property":{"hasTaskList":true,"hasIncompleteTasks":true},"location":{"placeholder":"Paris, Île-de-France, France","latitude":48.8566,"longitude":2.3522}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(13,'benbucketlist1',3,strftime('%s','now','-13 days'),strftime('%s','now','-13 days'),replace('## 🌍 My travel bucket list #travel/bucketlist\n\n> Writing this from a tiny café near the Seine. Will keep updating after each trip. 📍\n\n### Been\n- [x] Paris, France — the croissants ruined every croissant after\n- [x] Shanghai, China — modern skyline next to thousand-year-old temples, somehow it works\n- [x] Grand Canyon, USA — photos do not do it justice, and I say that professionally\n- [x] Barcelona, Spain — Gaudí''s buildings look grown, not built\n\n### Next\n- [ ] Northern Lights in Iceland — booking for winter, *finally*\n- [ ] Safari in Tanzania\n- [ ] Great Barrier Reef, before it gets worse\n- [ ] New Zealand road trip — south island, two weeks minimum','\n',char(10)),'PUBLIC','{"tags":["travel","travel/bucketlist"],"property":{"hasTaskList":true,"hasIncompleteTasks":true},"location":{"placeholder":"Paris, Île-de-France, France","latitude":48.8566,"longitude":2.3522}}');
|
||||
|
||||
-- 14. Alice: git cheat sheet — long-form anchor with code blocks
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(14,'alicegitcheat1',2,strftime('%s','now','-17 days'),strftime('%s','now','-17 days'),replace('## ⚡ Git commands I keep forgetting #dev/git #dev/cheatsheet\n\nWriting these down so I stop googling the same things every week.\n\n```bash\n# Show the last commit in detail\ngit show --stat HEAD\n\n# Undo the last commit, keep changes staged\ngit reset --soft HEAD~1\n\n# Stash including untracked files\ngit stash push -u -m "wip: before refactor"\n\n# Find which commit introduced a string\ngit log -S "function_name" --source --all\n```\n\nThe `-S` one is gold for archaeology — way better than scrolling blame.','\n',char(10)),'PUBLIC','{"tags":["dev/git","dev/cheatsheet"],"property":{"hasLink":false,"hasCode":true}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(14,'alicegitcheat1',2,strftime('%s','now','-17 days'),strftime('%s','now','-17 days'),replace('## ⚡ Git commands I keep forgetting #dev/git #dev/cheatsheet\n\nWriting these down so I stop googling the same things every week.\n\n```bash\n# Show the last commit in detail\ngit show --stat HEAD\n\n# Undo the last commit, keep changes staged\ngit reset --soft HEAD~1\n\n# Stash including untracked files\ngit stash push -u -m "wip: before refactor"\n\n# Find which commit introduced a string\ngit log -S "function_name" --source --all\n```\n\nThe `-S` one is gold for archaeology — way better than scrolling blame.','\n',char(10)),'PUBLIC','{"tags":["dev","dev/git","dev/cheatsheet"],"property":{"hasLink":false,"hasCode":true}}');
|
||||
|
||||
-- 15. Alice: reading note + café location (oldest memo — the back of the feed)
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(15,'alicereading01',2,strftime('%s','now','-22 days'),strftime('%s','now','-22 days'),replace('## 📖 Reading: Deep Work #books\n\nStarted Cal Newport''s *Deep Work* this week. This passage stopped me:\n\n> "Human beings, it seems, are at their best when immersed deeply in something challenging."\n\nThat''s the whole pitch in one sentence. The rest of the book is evidence.\n\nExperiment for the next two weeks: no Slack or email before 11am. See what shifts when the morning is mine.','\n',char(10)),'PUBLIC','{"tags":["books"],"property":{"hasLink":false},"location":{"placeholder":"Sightglass Coffee, San Francisco, California, United States","latitude":37.7726,"longitude":-122.4099}}');
|
||||
|
|
@ -93,10 +93,10 @@ INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,pay
|
|||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(16,'protectedintro',1,strftime('%s','now','-20 hours'),strftime('%s','now','-20 hours'),replace('🔓 **If you can read this, you''re signed in.**\n\nThis memo''s visibility is **Protected** — every member of this instance can see it, but the logged-out public can''t. There''s a third level, **Private**, that only the author ever sees.\n\nEvery memo picks its own level: public for the world, protected for your team or community, private for your 3am thoughts. #getting-started','\n',char(10)),'PROTECTED','{"tags":["getting-started"],"property":{"hasLink":false}}');
|
||||
|
||||
-- 17. Alice: half-formed idea — protected memos are naturally rougher
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(17,'alicedigestcli',2,strftime('%s','now','-9 hours'),strftime('%s','now','-9 hours'),'Half-formed idea: a tiny CLI that collects my #dev/til memos into a weekly digest. Memos has a full API with access tokens, so this is one evening of work. Parking it here so future me has no excuses.','PROTECTED','{"tags":["dev/til"],"property":{"hasLink":false}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(17,'alicedigestcli',2,strftime('%s','now','-9 hours'),strftime('%s','now','-9 hours'),'Half-formed idea: a tiny CLI that collects my #dev/til memos into a weekly digest. Memos has a full API with access tokens, so this is one evening of work. Parking it here so future me has no excuses.','PROTECTED','{"tags":["dev","dev/til"],"property":{"hasLink":false}}');
|
||||
|
||||
-- 18. Ben: members-only packing list
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(18,'benpacklist001',3,strftime('%s','now','-5 days'),strftime('%s','now','-5 days'),replace('Coast trip packing list — members-only edition, because nobody needs to know how much coffee gear I carry:\n\n- [x] Camera + the one lens I''ll actually use\n- [x] AeroPress, hand grinder, scale ☕\n- [x] Rain shell\n- [ ] The second lens I won''t use but will pack anyway\n\n#travel/hikes','\n',char(10)),'PROTECTED','{"tags":["travel/hikes"],"property":{"hasLink":false,"hasTaskList":true,"hasIncompleteTasks":true}}');
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(18,'benpacklist001',3,strftime('%s','now','-5 days'),strftime('%s','now','-5 days'),replace('Coast trip packing list — members-only edition, because nobody needs to know how much coffee gear I carry:\n\n- [x] Camera + the one lens I''ll actually use\n- [x] AeroPress, hand grinder, scale ☕\n- [x] Rain shell\n- [ ] The second lens I won''t use but will pack anyway\n\n#travel/hikes','\n',char(10)),'PROTECTED','{"tags":["travel","travel/hikes"],"property":{"hasLink":false,"hasTaskList":true,"hasIncompleteTasks":true}}');
|
||||
|
||||
-- 19. Zoe: confession
|
||||
INSERT INTO memo (id,uid,creator_id,created_ts,updated_ts,content,visibility,payload) VALUES(19,'zoebookpile001',4,strftime('%s','now','-30 hours'),strftime('%s','now','-30 hours'),'Note to self: stop buying books faster than I read them. Current pile: 9. Keeping this memo protected so only fellow members witness my shame 📚 #books','PROTECTED','{"tags":["books"],"property":{"hasLink":false}}');
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
package test
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
|
|
@ -107,12 +108,12 @@ func TestMemoFilterTagsAll(t *testing.T) {
|
|||
tc := NewMemoFilterTestContext(t)
|
||||
defer tc.Close()
|
||||
|
||||
tc.CreateMemo(NewMemoBuilder("memo-all-work", tc.User.ID).Content("all work").Tags("work/a", "work/b"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Content("mixed").Tags("work/a", "home"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-all-work", tc.User.ID).Content("all work").Tags("work", "work/a", "work/b"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Content("mixed").Tags("work", "work/a", "home"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-untagged", tc.User.ID).Content("untagged"))
|
||||
|
||||
// Every tag starts with "work/": only the all-work memo qualifies.
|
||||
memos := tc.ListWithFilter(`tags.all(t, t.startsWith("work/"))`)
|
||||
// Every tag belongs to the work hierarchy: only the all-work memo qualifies.
|
||||
memos := tc.ListWithFilter(`tags.all(t, t.startsWith("work"))`)
|
||||
require.Len(t, memos, 1)
|
||||
require.Equal(t, "memo-all-work", memos[0].UID)
|
||||
|
||||
|
|
@ -125,12 +126,23 @@ func TestMemoFilterTagsAllEquals(t *testing.T) {
|
|||
tc := NewMemoFilterTestContext(t)
|
||||
defer tc.Close()
|
||||
|
||||
tc.CreateMemo(NewMemoBuilder("memo-only-x", tc.User.ID).Content("only x").Tags("x", "x"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-only-x", tc.User.ID).Content("only x").Tags("x"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-x-and-y", tc.User.ID).Content("x and y").Tags("x", "y"))
|
||||
|
||||
memos := tc.ListWithFilter(`tags.all(t, t == "x")`)
|
||||
require.Len(t, memos, 1)
|
||||
require.Equal(t, "memo-only-x", memos[0].UID)
|
||||
tc.CreateMemo(NewMemoBuilder("memo-upper-work", tc.User.ID).Tags("Work"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-lower-work", tc.User.ID).Tags("work"))
|
||||
memos = tc.ListWithFilter(`tags.all(t, t == "work")`)
|
||||
require.Len(t, memos, 1)
|
||||
require.Equal(t, "memo-lower-work", memos[0].UID)
|
||||
|
||||
longTag := strings.Repeat("a", 513)
|
||||
tc.CreateMemo(NewMemoBuilder("memo-long-tag", tc.User.ID).Tags(longTag))
|
||||
memos = tc.ListWithFilter(`tags.all(t, t == "` + longTag + `")`)
|
||||
require.Len(t, memos, 1)
|
||||
require.Equal(t, "memo-long-tag", memos[0].UID)
|
||||
}
|
||||
|
||||
func TestMemoFilterContentStartsWithEscaping(t *testing.T) {
|
||||
|
|
@ -199,16 +211,16 @@ func TestMemoFilterTagsAllMorePredicates(t *testing.T) {
|
|||
tc := NewMemoFilterTestContext(t)
|
||||
defer tc.Close()
|
||||
|
||||
tc.CreateMemo(NewMemoBuilder("memo-all-done", tc.User.ID).Tags("a/done", "b/done"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Tags("a/done", "b/todo"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-single", tc.User.ID).Tags("solo/done"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-all-done", tc.User.ID).Tags("done", "well-done"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Tags("done", "todo"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-single", tc.User.ID).Tags("solo-done"))
|
||||
|
||||
// endsWith: every tag ends with "done".
|
||||
memos := tc.ListWithFilter(`tags.all(t, t.endsWith("done"))`)
|
||||
require.ElementsMatch(t, []string{"memo-all-done", "memo-single"}, uids(memos))
|
||||
|
||||
// contains: every tag contains "/".
|
||||
memos = tc.ListWithFilter(`tags.all(t, t.contains("/"))`)
|
||||
// contains: every tag contains "o".
|
||||
memos = tc.ListWithFilter(`tags.all(t, t.contains("o"))`)
|
||||
require.ElementsMatch(t, []string{"memo-all-done", "memo-mixed", "memo-single"}, uids(memos))
|
||||
}
|
||||
|
||||
|
|
@ -217,17 +229,17 @@ func TestMemoFilterTagsAllNegatedAndCombined(t *testing.T) {
|
|||
tc := NewMemoFilterTestContext(t)
|
||||
defer tc.Close()
|
||||
|
||||
tc.CreateMemo(NewMemoBuilder("memo-work", tc.User.ID).Content("work memo").Tags("work/a", "work/b"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Content("mixed memo").Tags("work/a", "home"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-work", tc.User.ID).Content("work memo").Tags("work", "work/a", "work/b"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Content("mixed memo").Tags("work", "work/a", "home"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-untagged", tc.User.ID).Content("untagged memo"))
|
||||
|
||||
// Negation: NOT all-work. The untagged memo's all() is false (non-empty guard),
|
||||
// so !all() is true and it is included.
|
||||
memos := tc.ListWithFilter(`!tags.all(t, t.startsWith("work/"))`)
|
||||
memos := tc.ListWithFilter(`!tags.all(t, t.startsWith("work"))`)
|
||||
require.ElementsMatch(t, []string{"memo-mixed", "memo-untagged"}, uids(memos))
|
||||
|
||||
// Combined with a content filter.
|
||||
memos = tc.ListWithFilter(`tags.all(t, t.startsWith("work/")) && content.contains("work")`)
|
||||
memos = tc.ListWithFilter(`tags.all(t, t.startsWith("work")) && content.contains("work")`)
|
||||
require.Len(t, memos, 1)
|
||||
require.Equal(t, "memo-work", memos[0].UID)
|
||||
}
|
||||
|
|
@ -489,13 +501,69 @@ func TestMemoFilterHierarchicalTags(t *testing.T) {
|
|||
defer tc.Close()
|
||||
|
||||
tc.CreateMemo(NewMemoBuilder("memo-book", tc.User.ID).Content("Book memo").Tags("book"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-book-fiction", tc.User.ID).Content("Fiction book memo").Tags("book/fiction"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-book-fiction", tc.User.ID).Content("Fiction book memo").Tags("book", "book/fiction"))
|
||||
|
||||
// Test: tag in ["book"] should match both (hierarchical matching)
|
||||
// Implied ancestors are stored in the memo tag set, so exact membership matches both memos.
|
||||
memos := tc.ListWithFilter(`tag in ["book"]`)
|
||||
require.Len(t, memos, 2)
|
||||
}
|
||||
|
||||
func TestMemoFilterTagMembershipIsExact(t *testing.T) {
|
||||
t.Parallel()
|
||||
tc := NewMemoFilterTestContext(t)
|
||||
defer tc.Close()
|
||||
const (
|
||||
composedTag = "caf\u00e9"
|
||||
decomposedTag = "cafe\u0301"
|
||||
)
|
||||
|
||||
tc.CreateMemo(NewMemoBuilder("memo-underscore", tc.User.ID).Tags("a_b"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-letter", tc.User.ID).Tags("axb"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-percent", tc.User.ID).Tags("a%b"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-letters", tc.User.ID).Tags("aZZb"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-upper", tc.User.ID).Tags("Work"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-lower", tc.User.ID).Tags("work"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-composed", tc.User.ID).Tags(composedTag))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-decomposed", tc.User.ID).Tags(decomposedTag))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-book", tc.User.ID).Tags("book"))
|
||||
tc.CreateMemo(NewMemoBuilder("memo-descendant-only", tc.User.ID).Tags("book/fiction"))
|
||||
|
||||
for _, test := range []struct {
|
||||
filter string
|
||||
want string
|
||||
}{
|
||||
{filter: `tag in ["a_b"]`, want: "memo-underscore"},
|
||||
{filter: `tag in ["a%b"]`, want: "memo-percent"},
|
||||
{filter: `tag in ["Work"]`, want: "memo-upper"},
|
||||
{filter: `tag in ["work"]`, want: "memo-lower"},
|
||||
{filter: `tag in ["` + composedTag + `"]`, want: "memo-composed"},
|
||||
{filter: `tag in ["` + decomposedTag + `"]`, want: "memo-decomposed"},
|
||||
{filter: `tag in ["book"]`, want: "memo-book"},
|
||||
{filter: `"work" in tags`, want: "memo-lower"},
|
||||
} {
|
||||
memos := tc.ListWithFilter(test.filter)
|
||||
require.Len(t, memos, 1, test.filter)
|
||||
require.Equal(t, test.want, memos[0].UID, test.filter)
|
||||
}
|
||||
|
||||
for _, test := range []struct {
|
||||
filter string
|
||||
want string
|
||||
}{
|
||||
{filter: `tags.exists(t, t.startsWith("a_"))`, want: "memo-underscore"},
|
||||
{filter: `tags.exists(t, t.endsWith("%b"))`, want: "memo-percent"},
|
||||
{filter: `tags.exists(t, t.contains("a%b"))`, want: "memo-percent"},
|
||||
{filter: `tags.all(t, t.startsWith("a_"))`, want: "memo-underscore"},
|
||||
{filter: `tags.exists_one(t, t.contains("a%b"))`, want: "memo-percent"},
|
||||
} {
|
||||
memos := tc.ListWithFilter(test.filter)
|
||||
require.Len(t, memos, 1, test.filter)
|
||||
require.Equal(t, test.want, memos[0].UID, test.filter)
|
||||
}
|
||||
|
||||
require.Empty(t, tc.ListWithFilter(`tags.exists(t, t.contains("A_"))`))
|
||||
}
|
||||
|
||||
func TestMemoFilterEmptyTags(t *testing.T) {
|
||||
t.Parallel()
|
||||
tc := NewMemoFilterTestContext(t)
|
||||
|
|
|
|||
|
|
@ -48,6 +48,8 @@
|
|||
"mdast-util-from-markdown": "^2.0.3",
|
||||
"mdast-util-gfm": "^3.1.0",
|
||||
"mermaid": "^11.14.0",
|
||||
"micromark-util-decode-string": "^2.0.1",
|
||||
"micromark-util-normalize-identifier": "^2.0.1",
|
||||
"micromark-extension-gfm": "^3.0.0",
|
||||
"mime": "^4.1.0",
|
||||
"react": "^19.2.6",
|
||||
|
|
|
|||
92
web/patches/@bufbuild__protobuf@2.12.0.patch
Normal file
92
web/patches/@bufbuild__protobuf@2.12.0.patch
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
diff --git a/dist/cjs/reflect/reflect.js b/dist/cjs/reflect/reflect.js
|
||||
--- a/dist/cjs/reflect/reflect.js
|
||||
+++ b/dist/cjs/reflect/reflect.js
|
||||
@@ -236,7 +236,19 @@ class ReflectMapImpl {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
- this.obj[mapKeyToLocal(key)] = mapValueToLocal(this._field, value);
|
||||
+ const localKey = mapKeyToLocal(key);
|
||||
+ const localValue = mapValueToLocal(this._field, value);
|
||||
+ if (localKey === "__proto__") {
|
||||
+ Object.defineProperty(this.obj, localKey, {
|
||||
+ value: localValue,
|
||||
+ writable: true,
|
||||
+ enumerable: true,
|
||||
+ configurable: true,
|
||||
+ });
|
||||
+ }
|
||||
+ else {
|
||||
+ this.obj[localKey] = localValue;
|
||||
+ }
|
||||
return this;
|
||||
}
|
||||
delete(key) {
|
||||
diff --git a/dist/cjs/create.js b/dist/cjs/create.js
|
||||
--- a/dist/cjs/create.js
|
||||
+++ b/dist/cjs/create.js
|
||||
@@ -138,6 +138,17 @@ function convertObjectValues(obj, fn) {
|
||||
const ret = {};
|
||||
for (const entry of Object.entries(obj)) {
|
||||
- ret[entry[0]] = fn(entry[1]);
|
||||
+ const value = fn(entry[1]);
|
||||
+ if (entry[0] === "__proto__") {
|
||||
+ Object.defineProperty(ret, entry[0], {
|
||||
+ value,
|
||||
+ writable: true,
|
||||
+ enumerable: true,
|
||||
+ configurable: true,
|
||||
+ });
|
||||
+ }
|
||||
+ else {
|
||||
+ ret[entry[0]] = value;
|
||||
+ }
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
diff --git a/dist/esm/create.js b/dist/esm/create.js
|
||||
--- a/dist/esm/create.js
|
||||
+++ b/dist/esm/create.js
|
||||
@@ -135,6 +135,17 @@ function convertObjectValues(obj, fn) {
|
||||
const ret = {};
|
||||
for (const entry of Object.entries(obj)) {
|
||||
- ret[entry[0]] = fn(entry[1]);
|
||||
+ const value = fn(entry[1]);
|
||||
+ if (entry[0] === "__proto__") {
|
||||
+ Object.defineProperty(ret, entry[0], {
|
||||
+ value,
|
||||
+ writable: true,
|
||||
+ enumerable: true,
|
||||
+ configurable: true,
|
||||
+ });
|
||||
+ }
|
||||
+ else {
|
||||
+ ret[entry[0]] = value;
|
||||
+ }
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
diff --git a/dist/esm/reflect/reflect.js b/dist/esm/reflect/reflect.js
|
||||
--- a/dist/esm/reflect/reflect.js
|
||||
+++ b/dist/esm/reflect/reflect.js
|
||||
@@ -229,7 +229,19 @@ class ReflectMapImpl {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
- this.obj[mapKeyToLocal(key)] = mapValueToLocal(this._field, value);
|
||||
+ const localKey = mapKeyToLocal(key);
|
||||
+ const localValue = mapValueToLocal(this._field, value);
|
||||
+ if (localKey === "__proto__") {
|
||||
+ Object.defineProperty(this.obj, localKey, {
|
||||
+ value: localValue,
|
||||
+ writable: true,
|
||||
+ enumerable: true,
|
||||
+ configurable: true,
|
||||
+ });
|
||||
+ }
|
||||
+ else {
|
||||
+ this.obj[localKey] = localValue;
|
||||
+ }
|
||||
return this;
|
||||
}
|
||||
delete(key) {
|
||||
|
|
@ -4,6 +4,9 @@ settings:
|
|||
autoInstallPeers: true
|
||||
excludeLinksFromLockfile: false
|
||||
|
||||
patchedDependencies:
|
||||
'@bufbuild/protobuf@2.12.0': 9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c
|
||||
|
||||
importers:
|
||||
|
||||
.:
|
||||
|
|
@ -31,10 +34,10 @@ importers:
|
|||
version: 6.43.6
|
||||
'@connectrpc/connect':
|
||||
specifier: ^2.1.1
|
||||
version: 2.1.1(@bufbuild/protobuf@2.12.0)
|
||||
version: 2.1.1(@bufbuild/protobuf@2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c))
|
||||
'@connectrpc/connect-web':
|
||||
specifier: ^2.1.1
|
||||
version: 2.1.1(@bufbuild/protobuf@2.12.0)(@connectrpc/connect@2.1.1(@bufbuild/protobuf@2.12.0))
|
||||
version: 2.1.1(@bufbuild/protobuf@2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c))(@connectrpc/connect@2.1.1(@bufbuild/protobuf@2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c)))
|
||||
'@github/relative-time-element':
|
||||
specifier: ^5.0.0
|
||||
version: 5.0.0
|
||||
|
|
@ -104,6 +107,12 @@ importers:
|
|||
micromark-extension-gfm:
|
||||
specifier: ^3.0.0
|
||||
version: 3.0.0
|
||||
micromark-util-decode-string:
|
||||
specifier: ^2.0.1
|
||||
version: 2.0.1
|
||||
micromark-util-normalize-identifier:
|
||||
specifier: ^2.0.1
|
||||
version: 2.0.1
|
||||
mime:
|
||||
specifier: ^4.1.0
|
||||
version: 4.1.0
|
||||
|
|
@ -176,7 +185,7 @@ importers:
|
|||
version: 2.4.14
|
||||
'@bufbuild/protobuf':
|
||||
specifier: ^2.12.0
|
||||
version: 2.12.0
|
||||
version: 2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c)
|
||||
'@rolldown/plugin-babel':
|
||||
specifier: 0.2.3
|
||||
version: 0.2.3(@babel/core@7.29.0)(@babel/plugin-transform-runtime@7.29.0(@babel/core@7.29.0))(@babel/runtime@7.29.2)(rolldown@1.0.0-rc.18)(vite@8.0.11(@types/node@24.12.2)(jiti@2.6.1)(terser@5.47.1))
|
||||
|
|
@ -2875,7 +2884,7 @@ snapshots:
|
|||
dependencies:
|
||||
css-tree: 3.2.1
|
||||
|
||||
'@bufbuild/protobuf@2.12.0': {}
|
||||
'@bufbuild/protobuf@2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c)': {}
|
||||
|
||||
'@chevrotain/cst-dts-gen@12.0.0':
|
||||
dependencies:
|
||||
|
|
@ -2972,14 +2981,14 @@ snapshots:
|
|||
style-mod: 4.1.3
|
||||
w3c-keyname: 2.2.8
|
||||
|
||||
'@connectrpc/connect-web@2.1.1(@bufbuild/protobuf@2.12.0)(@connectrpc/connect@2.1.1(@bufbuild/protobuf@2.12.0))':
|
||||
'@connectrpc/connect-web@2.1.1(@bufbuild/protobuf@2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c))(@connectrpc/connect@2.1.1(@bufbuild/protobuf@2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c)))':
|
||||
dependencies:
|
||||
'@bufbuild/protobuf': 2.12.0
|
||||
'@connectrpc/connect': 2.1.1(@bufbuild/protobuf@2.12.0)
|
||||
'@bufbuild/protobuf': 2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c)
|
||||
'@connectrpc/connect': 2.1.1(@bufbuild/protobuf@2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c))
|
||||
|
||||
'@connectrpc/connect@2.1.1(@bufbuild/protobuf@2.12.0)':
|
||||
'@connectrpc/connect@2.1.1(@bufbuild/protobuf@2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c))':
|
||||
dependencies:
|
||||
'@bufbuild/protobuf': 2.12.0
|
||||
'@bufbuild/protobuf': 2.12.0(patch_hash=9d60cec35402ba36bc21a72fefc68ccef8db0acb883504cc47c05a00a0029f8c)
|
||||
|
||||
'@csstools/color-helpers@6.0.2': {}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,2 +1,4 @@
|
|||
allowBuilds:
|
||||
esbuild: true
|
||||
patchedDependencies:
|
||||
"@bufbuild/protobuf@2.12.0": patches/@bufbuild__protobuf@2.12.0.patch
|
||||
|
|
|
|||
122
web/scripts/generate-tag-unicode-data.mjs
Normal file
122
web/scripts/generate-tag-unicode-data.mjs
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
import { createHash } from "node:crypto";
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const UNICODE_RELEASE = "17.0.0";
|
||||
const sources = {
|
||||
derived: `https://www.unicode.org/Public/${UNICODE_RELEASE}/ucd/DerivedCoreProperties.txt`,
|
||||
emoji: `https://www.unicode.org/Public/${UNICODE_RELEASE}/emoji/emoji-test.txt`,
|
||||
unicodeData: `https://www.unicode.org/Public/${UNICODE_RELEASE}/ucd/UnicodeData.txt`,
|
||||
};
|
||||
|
||||
const fetchText = async (url) => {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) throw new Error(`Failed to fetch ${url}: ${response.status}`);
|
||||
return response.text();
|
||||
};
|
||||
|
||||
const parseRange = (value) => {
|
||||
const [from, to = from] = value.split("..").map((part) => Number.parseInt(part, 16));
|
||||
return [from, to];
|
||||
};
|
||||
|
||||
const mergeRanges = (ranges) => {
|
||||
const merged = [];
|
||||
for (const [from, to] of ranges.sort((left, right) => left[0] - right[0])) {
|
||||
const previous = merged.at(-1);
|
||||
if (previous && from <= previous[1] + 1) previous[1] = Math.max(previous[1], to);
|
||||
else merged.push([from, to]);
|
||||
}
|
||||
return merged;
|
||||
};
|
||||
|
||||
const parseDerivedProperty = (source, property) =>
|
||||
mergeRanges(
|
||||
source
|
||||
.split("\n")
|
||||
.map((line) => line.match(/^([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/))
|
||||
.filter((match) => match?.[2] === property)
|
||||
.map((match) => parseRange(match[1])),
|
||||
);
|
||||
|
||||
const parseCombiningMarkRanges = (source) => {
|
||||
const ranges = [];
|
||||
let pending;
|
||||
for (const line of source.split("\n")) {
|
||||
if (!line) continue;
|
||||
const fields = line.split(";");
|
||||
const codePoint = Number.parseInt(fields[0], 16);
|
||||
const name = fields[1];
|
||||
const category = fields[2];
|
||||
if (name.endsWith(", First>")) {
|
||||
pending = { codePoint, category };
|
||||
} else if (name.endsWith(", Last>") && pending) {
|
||||
if (pending.category === "Mn" || pending.category === "Mc") ranges.push([pending.codePoint, codePoint]);
|
||||
pending = undefined;
|
||||
} else if (category === "Mn" || category === "Mc") {
|
||||
ranges.push([codePoint, codePoint]);
|
||||
}
|
||||
}
|
||||
return mergeRanges(ranges);
|
||||
};
|
||||
|
||||
const parseEmoji = (source) =>
|
||||
source
|
||||
.split("\n")
|
||||
.map((line) => line.match(/^([0-9A-F ]+)\s*;\s*fully-qualified\b/))
|
||||
.filter(Boolean)
|
||||
.map((match) =>
|
||||
String.fromCodePoint(
|
||||
...match[1]
|
||||
.trim()
|
||||
.split(/\s+/)
|
||||
.map((part) => Number.parseInt(part, 16)),
|
||||
),
|
||||
);
|
||||
|
||||
const formatRanges = (name, ranges) => {
|
||||
const values = ranges.flatMap(([from, to]) => [`0x${from.toString(16)}`, `0x${to.toString(16)}`]);
|
||||
const lines = [];
|
||||
for (let index = 0; index < values.length; index += 12) lines.push(` ${values.slice(index, index + 12).join(", ")},`);
|
||||
return `export const ${name}: readonly number[] = [\n${lines.join("\n")}\n];`;
|
||||
};
|
||||
|
||||
const formatEmoji = (emoji) => {
|
||||
const lines = [];
|
||||
for (let index = 0; index < emoji.length; index += 12) {
|
||||
lines.push(
|
||||
` ${emoji
|
||||
.slice(index, index + 12)
|
||||
.map((value) => JSON.stringify(value))
|
||||
.join(", ")},`,
|
||||
);
|
||||
}
|
||||
return `export const FULLY_QUALIFIED_EMOJI: readonly string[] = [\n${lines.join("\n")}\n];`;
|
||||
};
|
||||
|
||||
const unicodeDataDigest = (xidContinue, defaultIgnorable, combiningMarks, emoji) => {
|
||||
const canonical = [];
|
||||
const addRanges = (name, ranges) => {
|
||||
for (const [from, to] of ranges) canonical.push(`${name}:${from.toString(16)}-${to.toString(16)}\n`);
|
||||
};
|
||||
addRanges("xid", xidContinue);
|
||||
addRanges("ignorable", defaultIgnorable);
|
||||
addRanges("combining", combiningMarks);
|
||||
canonical.push(
|
||||
...emoji.map((value) => `emoji:${[...value].map((character) => character.codePointAt(0).toString(16)).join(",")}\n`).sort(),
|
||||
);
|
||||
return createHash("sha256").update(canonical.join("")).digest("hex");
|
||||
};
|
||||
|
||||
const [derived, emojiTest, unicodeData] = await Promise.all(Object.values(sources).map(fetchText));
|
||||
const xidContinue = parseDerivedProperty(derived, "XID_Continue");
|
||||
const defaultIgnorable = parseDerivedProperty(derived, "Default_Ignorable_Code_Point");
|
||||
const combiningMarks = parseCombiningMarkRanges(unicodeData);
|
||||
const emoji = parseEmoji(emojiTest);
|
||||
const output = `// Generated by scripts/generate-tag-unicode-data.mjs. Do not edit.\n// Unicode and Emoji ${UNICODE_RELEASE} data used by Memos tag recognition.\n// Unicode data SHA-256: ${unicodeDataDigest(xidContinue, defaultIgnorable, combiningMarks, emoji)}\n// biome-ignore-all format: generated data is kept in compact source order\n\n${formatRanges(
|
||||
"XID_CONTINUE_RANGES",
|
||||
xidContinue,
|
||||
)}\n\n${formatRanges("DEFAULT_IGNORABLE_RANGES", defaultIgnorable)}\n\n${formatRanges("COMBINING_MARK_RANGES", combiningMarks)}\n\n${formatEmoji(emoji)}\n`;
|
||||
|
||||
const outputURL = new URL("../src/utils/tag-unicode-data.ts", import.meta.url);
|
||||
await writeFile(fileURLToPath(outputURL), output);
|
||||
|
|
@ -165,9 +165,9 @@ export const MemoMarkdownRendererCore = ({
|
|||
...mathRemarkPlugins,
|
||||
remarkGfm,
|
||||
remarkSplitMixedTaskLists,
|
||||
remarkBreaks,
|
||||
remarkMention,
|
||||
remarkTag,
|
||||
remarkMention,
|
||||
remarkBreaks,
|
||||
remarkPreserveType,
|
||||
]}
|
||||
rehypePlugins={[rehypeRaw, [rehypeSanitize, SANITIZE_SCHEMA], rehypeHeadingId, ...mathRehypePlugins]}
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ import { markdown } from "@codemirror/lang-markdown";
|
|||
import { indentUnit } from "@codemirror/language";
|
||||
import { Compartment, type Extension } from "@codemirror/state";
|
||||
import { placeholder as cmPlaceholder, dropCursor, EditorView, type KeyBinding, keymap } from "@codemirror/view";
|
||||
import { GFM } from "@lezer/markdown";
|
||||
import { memoMarkdownExtensions } from "@/utils/memo-markdown-extension";
|
||||
import { headingDecorations } from "./headingDecorations";
|
||||
import { liftListItem, sinkListItem } from "./listIndent";
|
||||
import { tagAutocomplete } from "./tagAutocomplete";
|
||||
|
|
@ -77,7 +77,7 @@ export function buildEditorExtensions({
|
|||
dropCursor(),
|
||||
// Indent with spaces (markdown), matching the 2-space bullet nesting.
|
||||
indentUnit.of(" "),
|
||||
markdown({ extensions: [GFM] }),
|
||||
markdown({ extensions: memoMarkdownExtensions }),
|
||||
...memoEditorTheme,
|
||||
EditorView.lineWrapping,
|
||||
// CodeMirror defaults to autocorrect="off" because it is primarily a code
|
||||
|
|
|
|||
339
web/src/components/MemoEditor/Editor/markdownTagRanges.ts
Normal file
339
web/src/components/MemoEditor/Editor/markdownTagRanges.ts
Normal file
|
|
@ -0,0 +1,339 @@
|
|||
import { ensureSyntaxTree, syntaxTree, syntaxTreeAvailable } from "@codemirror/language";
|
||||
import type { EditorState } from "@codemirror/state";
|
||||
import { parser as markdownParser } from "@lezer/markdown";
|
||||
import { fromMarkdown } from "mdast-util-from-markdown";
|
||||
import { decodeString } from "micromark-util-decode-string";
|
||||
import { findMarkdownGFMEmailRanges, type GFMEmailSourceRange, type MarkdownSourceNode } from "@/utils/gfm-email";
|
||||
import { findMarkdownGFMURLRanges } from "@/utils/gfm-url";
|
||||
import {
|
||||
hasExactMarkdownRange,
|
||||
isValidMarkdownLinkReference,
|
||||
type MarkdownSourceRange,
|
||||
resolvedMarkdownLinkRanges,
|
||||
} from "@/utils/markdown-link";
|
||||
import { findDecodedMarkdownSourceBoundaries } from "@/utils/markdown-source-boundaries";
|
||||
import { memoMarkdownExtensions } from "@/utils/memo-markdown-extension";
|
||||
import { findTagMatches, type TagMatch } from "@/utils/tag-grammar";
|
||||
|
||||
interface SourceRange {
|
||||
from: number;
|
||||
to: number;
|
||||
}
|
||||
|
||||
type MarkdownNode = MarkdownSourceNode;
|
||||
|
||||
interface ParsedMarkdownContext {
|
||||
parsedTo: number;
|
||||
source: string;
|
||||
root: MarkdownNode;
|
||||
resolvedLinks: MarkdownSourceRange[];
|
||||
writtenURLRanges: MarkdownSourceRange[];
|
||||
emailRanges: GFMEmailSourceRange[];
|
||||
allowUnresolvedLinks: boolean;
|
||||
}
|
||||
|
||||
const sourceParser = markdownParser.configure(memoMarkdownExtensions);
|
||||
const REFERENCE_LABEL_PREFIX = "x ";
|
||||
const REJECTED_URL_PREFIX = "x";
|
||||
const parsedContextCache = new WeakMap<EditorState, ParsedMarkdownContext>();
|
||||
let canonicalCodeCache: { source: string; ranges: SourceRange[] } | undefined;
|
||||
|
||||
function parsedMarkdownRoot(state: EditorState, to: number): MarkdownNode {
|
||||
return (ensureSyntaxTree(state, to) ?? syntaxTree(state)).topNode as MarkdownNode;
|
||||
}
|
||||
|
||||
function parsedMarkdownContext(state: EditorState, to: number): ParsedMarkdownContext {
|
||||
const cached = parsedContextCache.get(state);
|
||||
if (cached && cached.parsedTo >= to) return cached;
|
||||
|
||||
const source = state.doc.toString();
|
||||
const root = parsedMarkdownRoot(state, to);
|
||||
const resolvedLinks = resolvedMarkdownLinkRanges(source, root);
|
||||
const allowUnresolvedLinks = syntaxTreeAvailable(state);
|
||||
const context = {
|
||||
parsedTo: to,
|
||||
source,
|
||||
root,
|
||||
resolvedLinks,
|
||||
writtenURLRanges: findMarkdownGFMURLRanges(source, root, resolvedLinks),
|
||||
emailRanges: findMarkdownGFMEmailRanges(source, root, resolvedLinks, allowUnresolvedLinks),
|
||||
allowUnresolvedLinks,
|
||||
};
|
||||
parsedContextCache.set(state, context);
|
||||
return context;
|
||||
}
|
||||
|
||||
// Only these known textual containers expose their direct source gaps. Their
|
||||
// syntax children are boundaries: transparent children recurse, while links,
|
||||
// code, math, HTML syntax, escapes, entities, and future extension nodes are
|
||||
// opaque by default.
|
||||
const TEXT_CONTAINERS = new Set([
|
||||
"ATXHeading1",
|
||||
"ATXHeading2",
|
||||
"ATXHeading3",
|
||||
"ATXHeading4",
|
||||
"ATXHeading5",
|
||||
"ATXHeading6",
|
||||
"Emphasis",
|
||||
"Paragraph",
|
||||
"SetextHeading1",
|
||||
"SetextHeading2",
|
||||
"Strikethrough",
|
||||
"StrongEmphasis",
|
||||
"TableCell",
|
||||
"Task",
|
||||
]);
|
||||
|
||||
const STRUCTURAL_CONTAINERS = new Set([
|
||||
"Blockquote",
|
||||
"BulletList",
|
||||
"Document",
|
||||
"ListItem",
|
||||
"OrderedList",
|
||||
"Table",
|
||||
"TableHeader",
|
||||
"TableRow",
|
||||
]);
|
||||
|
||||
function addClampedRange(ranges: SourceRange[], from: number, to: number, lowerBound: number, upperBound: number): void {
|
||||
const clampedFrom = Math.max(from, lowerBound);
|
||||
const clampedTo = Math.min(to, upperBound);
|
||||
if (clampedFrom < clampedTo) ranges.push({ from: clampedFrom, to: clampedTo });
|
||||
}
|
||||
|
||||
function collectUnresolvedReferenceLabelRanges(node: MarkdownNode, from: number, to: number, source: string, ranges: SourceRange[]): void {
|
||||
const contentFrom = node.from + 1;
|
||||
const contentTo = node.to - 1;
|
||||
if (contentFrom >= contentTo) return;
|
||||
|
||||
// Lezer keeps an unresolved second label as one opaque LinkLabel node. Parse
|
||||
// its contents again as ordinary inline Markdown so emphasis, escapes, and
|
||||
// other syntax retain their normal boundaries after link resolution fails.
|
||||
const prefixedSource = REFERENCE_LABEL_PREFIX + source.slice(contentFrom, contentTo);
|
||||
const root = sourceParser.parse(prefixedSource).topNode as MarkdownNode;
|
||||
const resolvedLinks = resolvedMarkdownLinkRanges(prefixedSource, root);
|
||||
const writtenURLs = findMarkdownGFMURLRanges(prefixedSource, root, resolvedLinks, false).filter(
|
||||
(range) => range.from !== REFERENCE_LABEL_PREFIX.length,
|
||||
);
|
||||
const labelRanges: SourceRange[] = [];
|
||||
collectLiteralRanges(
|
||||
root,
|
||||
REFERENCE_LABEL_PREFIX.length,
|
||||
prefixedSource.length,
|
||||
prefixedSource,
|
||||
resolvedLinks,
|
||||
writtenURLs,
|
||||
true,
|
||||
labelRanges,
|
||||
);
|
||||
for (const range of labelRanges) {
|
||||
const offset = contentFrom - REFERENCE_LABEL_PREFIX.length;
|
||||
addClampedRange(ranges, range.from + offset, range.to + offset, from, to);
|
||||
}
|
||||
}
|
||||
|
||||
function collectRejectedURLLiteralRanges(
|
||||
parent: MarkdownNode,
|
||||
node: MarkdownNode,
|
||||
from: number,
|
||||
to: number,
|
||||
source: string,
|
||||
ranges: SourceRange[],
|
||||
): void {
|
||||
const prefixedSource = REJECTED_URL_PREFIX + source.slice(node.from, node.to);
|
||||
const root = sourceParser.parse(prefixedSource).topNode as MarkdownNode;
|
||||
const resolvedLinks = resolvedMarkdownLinkRanges(prefixedSource, root);
|
||||
const urlRanges: SourceRange[] = [];
|
||||
collectLiteralRanges(
|
||||
root,
|
||||
REJECTED_URL_PREFIX.length,
|
||||
prefixedSource.length,
|
||||
prefixedSource,
|
||||
resolvedLinks,
|
||||
findMarkdownGFMURLRanges(prefixedSource, root, resolvedLinks, false),
|
||||
true,
|
||||
urlRanges,
|
||||
);
|
||||
|
||||
const offset = node.from - REJECTED_URL_PREFIX.length;
|
||||
const projected = urlRanges.map((range) => ({ from: range.from + offset, to: range.to + offset }));
|
||||
const emphasisMarks: SourceRange[] = [];
|
||||
if (source.slice(node.from, node.to).includes("_") && source.slice(parent.from, node.from).includes("_")) {
|
||||
const insertion = node.from - parent.from;
|
||||
const contextualSource = source.slice(parent.from, node.from) + REJECTED_URL_PREFIX + source.slice(node.from, node.to);
|
||||
const contentFrom = insertion + REJECTED_URL_PREFIX.length;
|
||||
sourceParser.parse(contextualSource).iterate({
|
||||
enter(mark) {
|
||||
if (mark.name === "EmphasisMark" && mark.from >= contentFrom) {
|
||||
emphasisMarks.push({
|
||||
from: mark.from + parent.from - REJECTED_URL_PREFIX.length,
|
||||
to: mark.to + parent.from - REJECTED_URL_PREFIX.length,
|
||||
});
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
for (const range of subtractRanges(projected, emphasisMarks)) {
|
||||
addClampedRange(ranges, range.from, range.to, from, to);
|
||||
}
|
||||
}
|
||||
|
||||
function canonicalSourceKeepsCodeOpaque(source: string, node: MarkdownNode): boolean {
|
||||
if (canonicalCodeCache?.source !== source) {
|
||||
const ranges: SourceRange[] = [];
|
||||
const collectRanges = (candidate: unknown): void => {
|
||||
if (!candidate || typeof candidate !== "object") return;
|
||||
const value = candidate as {
|
||||
type?: string;
|
||||
position?: { start?: { offset?: number }; end?: { offset?: number } };
|
||||
children?: unknown[];
|
||||
};
|
||||
const from = value.position?.start?.offset;
|
||||
const to = value.position?.end?.offset;
|
||||
if (value.type === "code" && from !== undefined && to !== undefined) ranges.push({ from, to });
|
||||
value.children?.forEach(collectRanges);
|
||||
};
|
||||
collectRanges(fromMarkdown(source));
|
||||
canonicalCodeCache = { source, ranges };
|
||||
}
|
||||
|
||||
return canonicalCodeCache.ranges.some((range) => range.from <= node.from && range.to >= node.to);
|
||||
}
|
||||
|
||||
function collectReparsedLiteralRanges(node: MarkdownNode, from: number, to: number, source: string, ranges: SourceRange[]): void {
|
||||
const prefix = "x ";
|
||||
const prefixedSource = prefix + source.slice(node.from, node.to);
|
||||
const root = sourceParser.parse(prefixedSource).topNode as MarkdownNode;
|
||||
const resolvedLinks = resolvedMarkdownLinkRanges(prefixedSource, root);
|
||||
const reparsed: SourceRange[] = [];
|
||||
collectLiteralRanges(
|
||||
root,
|
||||
prefix.length,
|
||||
prefixedSource.length,
|
||||
prefixedSource,
|
||||
resolvedLinks,
|
||||
findMarkdownGFMURLRanges(prefixedSource, root, resolvedLinks, false),
|
||||
true,
|
||||
reparsed,
|
||||
);
|
||||
const offset = node.from - prefix.length;
|
||||
for (const range of reparsed) addClampedRange(ranges, range.from + offset, range.to + offset, from, to);
|
||||
}
|
||||
|
||||
function collectLiteralRanges(
|
||||
node: MarkdownNode,
|
||||
from: number,
|
||||
to: number,
|
||||
source: string,
|
||||
resolvedLinks: MarkdownSourceRange[],
|
||||
writtenURLs: MarkdownSourceRange[],
|
||||
allowUnresolvedLinks: boolean,
|
||||
ranges: SourceRange[],
|
||||
): void {
|
||||
if (node.to <= from || node.from >= to) return;
|
||||
|
||||
const unresolvedLinkLike =
|
||||
allowUnresolvedLinks && (node.name === "Link" || node.name === "Image") && !hasExactMarkdownRange(resolvedLinks, node);
|
||||
const invalidLinkReference = node.name === "LinkReference" && !isValidMarkdownLinkReference(source, node);
|
||||
const scansDirectText = TEXT_CONTAINERS.has(node.name) || unresolvedLinkLike || invalidLinkReference;
|
||||
if (!scansDirectText && !STRUCTURAL_CONTAINERS.has(node.name)) return;
|
||||
|
||||
let cursor = node.from;
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) {
|
||||
if (child.name === "Entity") {
|
||||
const spelling = source.slice(child.from, child.to);
|
||||
if (decodeString(spelling) === spelling) continue;
|
||||
}
|
||||
if (child.to <= cursor) continue;
|
||||
if (scansDirectText) addClampedRange(ranges, cursor, child.from, from, to);
|
||||
if (child.from >= cursor) {
|
||||
if (node.name === "Blockquote" && child.name === "CodeBlock" && !canonicalSourceKeepsCodeOpaque(source, child)) {
|
||||
collectReparsedLiteralRanges(child, from, to, source, ranges);
|
||||
} else if (invalidLinkReference && child.name === "LinkLabel") {
|
||||
collectUnresolvedReferenceLabelRanges(child, from, to, source, ranges);
|
||||
} else if (child.name === "URL") {
|
||||
if (!hasExactMarkdownRange(writtenURLs, child)) collectRejectedURLLiteralRanges(node, child, from, to, source, ranges);
|
||||
} else if (unresolvedLinkLike && child.name === "LinkLabel") {
|
||||
collectUnresolvedReferenceLabelRanges(child, from, to, source, ranges);
|
||||
} else {
|
||||
collectLiteralRanges(child, from, to, source, resolvedLinks, writtenURLs, allowUnresolvedLinks, ranges);
|
||||
}
|
||||
}
|
||||
cursor = Math.max(cursor, child.to);
|
||||
}
|
||||
if (scansDirectText) addClampedRange(ranges, cursor, node.to, from, to);
|
||||
}
|
||||
|
||||
function subtractRanges(ranges: SourceRange[], opaqueRanges: SourceRange[]): SourceRange[] {
|
||||
return ranges.flatMap((range) => {
|
||||
const pieces: SourceRange[] = [];
|
||||
let cursor = range.from;
|
||||
for (const opaque of opaqueRanges) {
|
||||
if (opaque.to <= cursor || opaque.from >= range.to) continue;
|
||||
if (cursor < opaque.from) pieces.push({ from: cursor, to: Math.min(opaque.from, range.to) });
|
||||
cursor = Math.max(cursor, opaque.to);
|
||||
if (cursor >= range.to) break;
|
||||
}
|
||||
if (cursor < range.to) pieces.push({ from: cursor, to: range.to });
|
||||
return pieces;
|
||||
});
|
||||
}
|
||||
|
||||
function mergeSourceRanges(ranges: SourceRange[]): SourceRange[] {
|
||||
const sorted = [...ranges].sort((left, right) => left.from - right.from || left.to - right.to);
|
||||
const merged: SourceRange[] = [];
|
||||
for (const range of sorted) {
|
||||
const previous = merged.at(-1);
|
||||
if (previous && range.from <= previous.to) previous.to = Math.max(previous.to, range.to);
|
||||
else merged.push({ ...range });
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
export function markdownGFMEmailSourceRanges(state: EditorState, to = state.doc.length): GFMEmailSourceRange[] {
|
||||
return parsedMarkdownContext(state, to).emailRanges;
|
||||
}
|
||||
|
||||
export function literalTagSourceRanges(state: EditorState, from: number, to: number): SourceRange[] {
|
||||
const ranges: SourceRange[] = [];
|
||||
const context = parsedMarkdownContext(state, to);
|
||||
collectLiteralRanges(
|
||||
context.root,
|
||||
from,
|
||||
to,
|
||||
context.source,
|
||||
context.resolvedLinks,
|
||||
context.writtenURLRanges,
|
||||
context.allowUnresolvedLinks,
|
||||
ranges,
|
||||
);
|
||||
const decodedBoundaries = findDecodedMarkdownSourceBoundaries(context.source, from, to);
|
||||
return mergeSourceRanges(
|
||||
subtractRanges(subtractRanges(subtractRanges(ranges, decodedBoundaries), context.emailRanges), context.writtenURLRanges),
|
||||
);
|
||||
}
|
||||
|
||||
/** Find tags only in literal Markdown source exposed by the editor syntax tree. */
|
||||
export function findMarkdownTagMatches(state: EditorState, from: number, to: number): TagMatch[] {
|
||||
const matches: TagMatch[] = [];
|
||||
for (const range of literalTagSourceRanges(state, from, to)) {
|
||||
const source = state.doc.sliceString(range.from, range.to);
|
||||
for (const match of findTagMatches(source)) {
|
||||
matches.push({ ...match, from: range.from + match.from, to: range.from + match.to });
|
||||
}
|
||||
}
|
||||
return matches;
|
||||
}
|
||||
|
||||
/** Return the active tag whose recognized source span ends at the cursor. */
|
||||
export function tagMatchBefore(state: EditorState, position: number): TagMatch | undefined {
|
||||
const line = state.doc.lineAt(position);
|
||||
const matches = findMarkdownTagMatches(state, line.from, position);
|
||||
return matches.findLast((match) => match.to === position);
|
||||
}
|
||||
|
||||
/** Whether an offset is in a literal-source run (used for explicit `#` completion). */
|
||||
export function isLiteralTagPosition(state: EditorState, position: number): boolean {
|
||||
const line = state.doc.lineAt(position);
|
||||
return literalTagSourceRanges(state, line.from, position).some((range) => range.to === position);
|
||||
}
|
||||
|
|
@ -1,22 +1,22 @@
|
|||
import { autocompletion, type CompletionContext, type CompletionResult } from "@codemirror/autocomplete";
|
||||
import type { Extension } from "@codemirror/state";
|
||||
import { TAG_CHAR_CLASS } from "@/utils/tag-grammar";
|
||||
|
||||
const TAG_BEFORE = new RegExp(`#(${TAG_CHAR_CLASS}*)$`, "u");
|
||||
import { isTagIntroducerAt } from "@/utils/tag-grammar";
|
||||
import { isLiteralTagPosition, tagMatchBefore } from "./markdownTagRanges";
|
||||
|
||||
export function makeTagCompletionSource(getTags: () => string[]) {
|
||||
return (ctx: CompletionContext): CompletionResult | null => {
|
||||
const before = ctx.matchBefore(TAG_BEFORE);
|
||||
if (!before || (before.from === before.to && !ctx.explicit)) return null;
|
||||
const typed = before.text.slice(1).toLowerCase();
|
||||
// Require at least one character after `#` unless completion was explicitly
|
||||
// invoked, so a bare `#` doesn't pop the menu while typing.
|
||||
if (typed.length < 1 && !ctx.explicit) return null;
|
||||
const match = tagMatchBefore(ctx.state, ctx.pos);
|
||||
const source = ctx.state.doc.toString();
|
||||
const explicitBareIntroducer =
|
||||
ctx.explicit && ctx.pos > 0 && isTagIntroducerAt(source, ctx.pos - 1) && isLiteralTagPosition(ctx.state, ctx.pos);
|
||||
if (!match && !explicitBareIntroducer) return null;
|
||||
|
||||
const typed = (match?.value ?? "").toLowerCase();
|
||||
const options = getTags()
|
||||
.filter((tag) => tag.toLowerCase().startsWith(typed))
|
||||
.map((tag) => ({ label: tag, type: "keyword" }));
|
||||
if (options.length === 0) return null;
|
||||
return { from: before.from + 1, options };
|
||||
return { from: match ? match.from + 1 : ctx.pos, options };
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,28 +1,38 @@
|
|||
import { RangeSetBuilder } from "@codemirror/state";
|
||||
import { Decoration, type DecorationSet, EditorView } from "@codemirror/view";
|
||||
import { MENTION_RUN } from "@/utils/mention-grammar";
|
||||
import { TAG_RUN } from "@/utils/tag-grammar";
|
||||
import { findMarkdownTagMatches, markdownGFMEmailSourceRanges } from "./markdownTagRanges";
|
||||
import { viewportDecorations } from "./viewportDecorations";
|
||||
|
||||
const TAG_RE = new RegExp(`(^|[^\\p{L}\\p{N}])#(${TAG_RUN})`, "gu");
|
||||
const MENTION_RE = new RegExp(`(^|[^A-Za-z0-9])@(${MENTION_RUN})`, "gu");
|
||||
const tagMark = Decoration.mark({ class: "cm-memo-tag" });
|
||||
const mentionMark = Decoration.mark({ class: "cm-memo-mention" });
|
||||
|
||||
function build(view: EditorView): DecorationSet {
|
||||
const ranges: { from: number; to: number; deco: Decoration }[] = [];
|
||||
for (const { from, to } of view.visibleRanges) {
|
||||
const visibleLines = view.visibleRanges
|
||||
.map(({ from, to }) => ({ from: view.state.doc.lineAt(from).from, to: view.state.doc.lineAt(to).to }))
|
||||
.sort((left, right) => left.from - right.from);
|
||||
const scanRanges: Array<{ from: number; to: number }> = [];
|
||||
for (const range of visibleLines) {
|
||||
const previous = scanRanges.at(-1);
|
||||
if (previous && range.from <= previous.to) previous.to = Math.max(previous.to, range.to);
|
||||
else scanRanges.push({ ...range });
|
||||
}
|
||||
const emailRanges = markdownGFMEmailSourceRanges(view.state, scanRanges.at(-1)?.to ?? view.viewport.to);
|
||||
|
||||
for (const { from, to } of scanRanges) {
|
||||
for (const match of findMarkdownTagMatches(view.state, from, to)) {
|
||||
ranges.push({ from: match.from, to: match.to, deco: tagMark });
|
||||
}
|
||||
|
||||
const text = view.state.doc.sliceString(from, to);
|
||||
for (const [re, deco] of [
|
||||
[TAG_RE, tagMark],
|
||||
[MENTION_RE, mentionMark],
|
||||
] as const) {
|
||||
re.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = re.exec(text)) !== null) {
|
||||
const start = from + m.index + m[1].length; // skip the boundary char
|
||||
ranges.push({ from: start, to: start + 1 + m[2].length, deco });
|
||||
}
|
||||
MENTION_RE.lastIndex = 0;
|
||||
let match: RegExpExecArray | null;
|
||||
while ((match = MENTION_RE.exec(text)) !== null) {
|
||||
const start = from + match.index + match[1].length; // skip the boundary char
|
||||
if (emailRanges.some((range) => range.from <= start && start < range.to)) continue;
|
||||
ranges.push({ from: start, to: start + 1 + match[2].length, deco: mentionMark });
|
||||
}
|
||||
}
|
||||
ranges.sort((a, b) => a.from - b.from || a.to - b.to);
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import { syntaxTree } from "@codemirror/language";
|
||||
import { type DecorationSet, EditorView, ViewPlugin, type ViewUpdate } from "@codemirror/view";
|
||||
|
||||
/**
|
||||
|
|
@ -13,7 +14,7 @@ export function viewportDecorations(build: (view: EditorView) => DecorationSet)
|
|||
this.decorations = build(view);
|
||||
}
|
||||
update(u: ViewUpdate) {
|
||||
if (u.docChanged || u.viewportChanged) {
|
||||
if (u.docChanged || u.viewportChanged || syntaxTree(u.startState) !== syntaxTree(u.state)) {
|
||||
this.decorations = build(u.view);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ MemoEditor/
|
|||
│ ├── extensions.ts # buildEditorExtensions(): assembles the CM extension set
|
||||
│ ├── theme.ts # Syntax-highlight style + editor theme (CSS-var colors)
|
||||
│ ├── tagMentionDecorations.ts# ViewPlugin that decorates #tag / @mention spans
|
||||
│ ├── markdownTagRanges.ts # Markdown syntax-tree adapter for the shared tag scanner
|
||||
│ ├── tagAutocomplete.ts # CM autocompletion source for #tag
|
||||
│ ├── formatting.ts # FormattingController impl (toggle marks, headings, lists)
|
||||
│ └── controller.ts # EditorController impl over an EditorView
|
||||
|
|
@ -83,10 +84,12 @@ Uses `useReducer` + Context for predictable state transitions. All state changes
|
|||
|
||||
### Tags and mentions
|
||||
|
||||
`#tag` autocomplete and `#tag`/`@mention` decoration both reuse the shared grammar so the editor can't drift from the rest of the app:
|
||||
`#tag` autocomplete, decoration, and read-only rendering all use the shared scanner in `@/utils/tag-grammar`. The scanner owns tag syntax and Unicode/emoji recognition; each surface supplies only its Markdown context:
|
||||
|
||||
- `Editor/tagMentionDecorations.ts` is a `ViewPlugin` that scans the visible ranges and adds `cm-memo-tag` / `cm-memo-mention` marks, matching against `TAG_RUN` (`@/utils/tag-grammar`) and `MENTION_RUN` (`@/utils/mention-grammar`).
|
||||
- `Editor/tagAutocomplete.ts` is a CodeMirror autocompletion source for `#tag`, matching the in-progress token with `TAG_CHAR_CLASS` (`@/utils/tag-grammar`) and offering known tags (from `useTagCounts`).
|
||||
- `Editor/markdownTagRanges.ts` adapts the CodeMirror syntax tree into literal-source ranges, excluding links, code, math, raw HTML syntax, escapes, and entities before calling the shared scanner.
|
||||
- `Editor/tagMentionDecorations.ts` decorates the tag matches returned by that adapter; mention recognition remains separate.
|
||||
- `Editor/tagAutocomplete.ts` reuses the same adapter for the tag ending at the cursor and offers known tags from `useTagCounts`.
|
||||
- `@/utils/remark-plugins/remark-tag` is the read-only renderer's Markdown AST adapter to the same scanner.
|
||||
|
||||
### Services
|
||||
|
||||
|
|
|
|||
|
|
@ -80,7 +80,7 @@ const TagsSection = () => {
|
|||
() =>
|
||||
Object.keys(localTags)
|
||||
.sort()
|
||||
.map((name) => ({ name, count: tagCounts[name] ?? 0 })),
|
||||
.map((name) => ({ name, count: Object.hasOwn(tagCounts, name) ? tagCounts[name] : 0 })),
|
||||
[localTags, tagCounts],
|
||||
);
|
||||
|
||||
|
|
@ -113,7 +113,7 @@ const TagsSection = () => {
|
|||
const handleAddTag = () => {
|
||||
const name = newTagName.trim();
|
||||
if (!name) return;
|
||||
if (localTags[name] !== undefined) {
|
||||
if (Object.hasOwn(localTags, name)) {
|
||||
toast.error(t("setting.tags.tag-already-exists"));
|
||||
return;
|
||||
}
|
||||
|
|
@ -183,7 +183,7 @@ const TagsSection = () => {
|
|||
/>
|
||||
<datalist id="known-tags">
|
||||
{allKnownTags
|
||||
.filter((tag) => !localTags[tag])
|
||||
.filter((tag) => !Object.hasOwn(localTags, tag))
|
||||
.map((tag) => (
|
||||
<option key={tag} value={tag} />
|
||||
))}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import type { MemoExplorerContext } from "@/components/MemoExplorer";
|
|||
import { type MemoTimeBasis, useView } from "@/contexts/ViewContext";
|
||||
import useCurrentUser from "@/hooks/useCurrentUser";
|
||||
import { useAllUserStats, useUserStats } from "@/hooks/useUserQueries";
|
||||
import { mergeTagCounts } from "@/lib/tag";
|
||||
import { State } from "@/types/proto/api/v1/common_pb";
|
||||
import type { UserStats } from "@/types/proto/api/v1/user_service_pb";
|
||||
import type { StatisticsData } from "@/types/statistics";
|
||||
|
|
@ -60,14 +61,12 @@ export const useFilteredMemoStats = (options: UseFilteredMemoStatsOptions = {}):
|
|||
const data = useMemo(() => {
|
||||
const loading = isLoadingUserStats || isLoadingAllUserStats;
|
||||
let activityStats: Record<string, number> = {};
|
||||
let tagCount: Record<string, number> = {};
|
||||
let tagCount: Record<string, number> = mergeTagCounts();
|
||||
|
||||
if (context === "explore" || context === "archived") {
|
||||
const displayDates: string[] = [];
|
||||
tagCount = mergeTagCounts(...allUserStats.map((stats) => stats.tagCount));
|
||||
for (const stats of allUserStats) {
|
||||
for (const [tag, count] of Object.entries(stats.tagCount ?? {})) {
|
||||
tagCount[tag] = (tagCount[tag] ?? 0) + count;
|
||||
}
|
||||
displayDates.push(
|
||||
...timestampsForBasis(stats, timeBasis)
|
||||
.map((ts) => (ts ? timestampDate(ts) : undefined))
|
||||
|
|
@ -88,7 +87,7 @@ export const useFilteredMemoStats = (options: UseFilteredMemoStatsOptions = {}):
|
|||
);
|
||||
}
|
||||
if (userStats.tagCount) {
|
||||
tagCount = userStats.tagCount;
|
||||
tagCount = mergeTagCounts(userStats.tagCount);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import { queryOptions, useMutation, useQuery, useQueryClient } from "@tanstack/r
|
|||
import { shortcutServiceClient, userServiceClient } from "@/connect";
|
||||
import useCurrentUser from "@/hooks/useCurrentUser";
|
||||
import { buildUserSettingName, userNamePrefix } from "@/lib/resource-names";
|
||||
import { mergeTagCounts } from "@/lib/tag";
|
||||
import {
|
||||
type ListAllUserStatsRequest,
|
||||
ListAllUserStatsRequestSchema,
|
||||
|
|
@ -119,20 +120,12 @@ export function useTagCounts(forCurrentUser = false) {
|
|||
const { stats } = await userServiceClient.listAllUserStats({});
|
||||
|
||||
// Aggregate tag counts from all users
|
||||
const tagCount: Record<string, number> = {};
|
||||
for (const userStats of stats) {
|
||||
if (userStats.tagCount) {
|
||||
for (const [tag, count] of Object.entries(userStats.tagCount)) {
|
||||
tagCount[tag] = (tagCount[tag] || 0) + count;
|
||||
}
|
||||
}
|
||||
}
|
||||
return tagCount;
|
||||
return mergeTagCounts(...stats.map((userStats) => userStats.tagCount));
|
||||
}
|
||||
},
|
||||
select: (data) => {
|
||||
if (forCurrentUser) {
|
||||
return (data as UserStats).tagCount || {};
|
||||
return mergeTagCounts((data as UserStats).tagCount);
|
||||
}
|
||||
return data as Record<string, number>;
|
||||
},
|
||||
|
|
|
|||
|
|
@ -3,6 +3,15 @@ import type { UserSetting_TagMetadata, UserSetting_TagsSetting } from "@/types/p
|
|||
// Cache compiled regexes to avoid re-compiling on every tag render.
|
||||
const compiledPatternCache = new Map<string, RegExp | null>();
|
||||
|
||||
/** Merge exact tag counts without inheriting Object prototype keys. */
|
||||
export const mergeTagCounts = (...sources: Array<Record<string, number> | undefined>): Record<string, number> => {
|
||||
const result = Object.create(null) as Record<string, number>;
|
||||
for (const source of sources) {
|
||||
for (const [tag, count] of Object.entries(source ?? {})) result[tag] = (result[tag] ?? 0) + count;
|
||||
}
|
||||
return result;
|
||||
};
|
||||
|
||||
const getCompiledPattern = (pattern: string): RegExp | null => {
|
||||
if (compiledPatternCache.has(pattern)) {
|
||||
return compiledPatternCache.get(pattern)!;
|
||||
|
|
@ -27,7 +36,7 @@ const getCompiledPattern = (pattern: string): RegExp | null => {
|
|||
*/
|
||||
export const findTagMetadata = (tag: string, tagsSetting: UserSetting_TagsSetting): UserSetting_TagMetadata | undefined => {
|
||||
// Fast path: exact match.
|
||||
if (tagsSetting.tags[tag]) {
|
||||
if (Object.hasOwn(tagsSetting.tags, tag)) {
|
||||
return tagsSetting.tags[tag];
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -104,9 +104,9 @@ const shortcutExamples = [
|
|||
icon: FilterIcon,
|
||||
},
|
||||
{
|
||||
title: "All tags match",
|
||||
filter: 'tags.all(t, t.startsWith("work/"))',
|
||||
description: "Every tag must satisfy the predicate (tagged memos only).",
|
||||
title: "All tags complete",
|
||||
filter: 'tags.all(t, t.endsWith("-done"))',
|
||||
description: "Every tag ends in -done (tagged memos only).",
|
||||
icon: TagsIcon,
|
||||
},
|
||||
{
|
||||
|
|
|
|||
214
web/src/utils/gfm-email.ts
Normal file
214
web/src/utils/gfm-email.ts
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
import { parser as markdownParser } from "@lezer/markdown";
|
||||
import { decodeString } from "micromark-util-decode-string";
|
||||
import { isValidMarkdownLinkReference, resolvedMarkdownLinkRanges } from "@/utils/markdown-link";
|
||||
import { decodedMarkdownCharacterReferenceAt } from "@/utils/markdown-source-boundaries";
|
||||
import { memoMarkdownExtensions } from "@/utils/memo-markdown-extension";
|
||||
|
||||
export interface GFMEmailSourceRange {
|
||||
from: number;
|
||||
to: number;
|
||||
value: string;
|
||||
}
|
||||
|
||||
export interface MarkdownSourceNode {
|
||||
readonly name: string;
|
||||
readonly from: number;
|
||||
readonly to: number;
|
||||
readonly firstChild: MarkdownSourceNode | null;
|
||||
readonly nextSibling: MarkdownSourceNode | null;
|
||||
}
|
||||
|
||||
interface MarkdownSourceRange {
|
||||
from: number;
|
||||
to: number;
|
||||
}
|
||||
|
||||
const DOMAIN_AT_START = /^@(?:[A-Za-z0-9_-]+\.)+[A-Za-z0-9_-]*[A-Za-z0-9]/;
|
||||
const REFERENCE_LABEL_PREFIX = "x ";
|
||||
const sourceParser = markdownParser.configure(memoMarkdownExtensions);
|
||||
|
||||
const TEXT_CONTAINERS = new Set([
|
||||
"ATXHeading1",
|
||||
"ATXHeading2",
|
||||
"ATXHeading3",
|
||||
"ATXHeading4",
|
||||
"ATXHeading5",
|
||||
"ATXHeading6",
|
||||
"Emphasis",
|
||||
"Paragraph",
|
||||
"SetextHeading1",
|
||||
"SetextHeading2",
|
||||
"Strikethrough",
|
||||
"StrongEmphasis",
|
||||
"TableCell",
|
||||
"Task",
|
||||
]);
|
||||
|
||||
const STRUCTURAL_CONTAINERS = new Set([
|
||||
"Blockquote",
|
||||
"BulletList",
|
||||
"Document",
|
||||
"ListItem",
|
||||
"OrderedList",
|
||||
"Table",
|
||||
"TableHeader",
|
||||
"TableRow",
|
||||
]);
|
||||
|
||||
function isLocalChar(char: string): boolean {
|
||||
return /[A-Za-z0-9._+-]/.test(char);
|
||||
}
|
||||
|
||||
function isASCIIPunctuation(char: string): boolean {
|
||||
const code = char.charCodeAt(0);
|
||||
return (
|
||||
(code >= 0x21 && code <= 0x2f) || (code >= 0x3a && code <= 0x40) || (code >= 0x5b && code <= 0x60) || (code >= 0x7b && code <= 0x7e)
|
||||
);
|
||||
}
|
||||
|
||||
interface EmailProjection {
|
||||
value: string;
|
||||
starts: number[];
|
||||
ends: number[];
|
||||
}
|
||||
|
||||
function projectEmailText(source: string, lowerBound: number, upperBound: number): EmailProjection {
|
||||
let value = "";
|
||||
const starts: number[] = [];
|
||||
const ends: number[] = [];
|
||||
for (let cursor = lowerBound; cursor < upperBound; ) {
|
||||
if (source[cursor] === "\\" && cursor + 1 < upperBound && isASCIIPunctuation(source[cursor + 1])) {
|
||||
value += source[cursor + 1];
|
||||
starts.push(cursor);
|
||||
ends.push(cursor + 2);
|
||||
cursor += 2;
|
||||
continue;
|
||||
}
|
||||
|
||||
const reference = decodedMarkdownCharacterReferenceAt(source, cursor, upperBound);
|
||||
if (reference) {
|
||||
value += reference.value;
|
||||
for (let index = 0; index < reference.value.length; index++) {
|
||||
starts.push(cursor);
|
||||
ends.push(reference.to);
|
||||
}
|
||||
cursor = reference.to;
|
||||
continue;
|
||||
}
|
||||
|
||||
const codePoint = source.codePointAt(cursor);
|
||||
const character = codePoint === undefined ? "" : String.fromCodePoint(codePoint);
|
||||
value += character;
|
||||
for (let index = 0; index < character.length; index++) {
|
||||
starts.push(cursor);
|
||||
ends.push(cursor + character.length);
|
||||
}
|
||||
cursor += character.length;
|
||||
}
|
||||
return { value, starts, ends };
|
||||
}
|
||||
|
||||
function matchEmailAt(source: string, at: number, lowerBound: number, upperBound: number): GFMEmailSourceRange | undefined {
|
||||
let start = at;
|
||||
while (start > lowerBound) {
|
||||
if (!isLocalChar(source[start - 1])) break;
|
||||
start--;
|
||||
}
|
||||
if (start === at) return undefined;
|
||||
|
||||
const domain = source.slice(at, upperBound).match(DOMAIN_AT_START)?.[0];
|
||||
if (!domain) return undefined;
|
||||
const end = at + domain.length;
|
||||
if (end < upperBound && (source[end] === "-" || source[end] === "_")) return undefined;
|
||||
return { from: start, to: end, value: source.slice(start, end) };
|
||||
}
|
||||
|
||||
/** Find extended GFM email autolinks inside one parsed text-source run. */
|
||||
export function findGFMEmailMatches(source: string, lowerBound = 0, upperBound = source.length): GFMEmailSourceRange[] {
|
||||
const projection = projectEmailText(source, lowerBound, upperBound);
|
||||
const matches: GFMEmailSourceRange[] = [];
|
||||
for (let cursor = 0; cursor < projection.value.length; ) {
|
||||
const at = projection.value.indexOf("@", cursor);
|
||||
if (at < 0) break;
|
||||
const match = matchEmailAt(projection.value, at, cursor, projection.value.length);
|
||||
if (!match) {
|
||||
cursor = at + 1;
|
||||
continue;
|
||||
}
|
||||
matches.push({
|
||||
from: projection.starts[match.from],
|
||||
to: projection.ends[match.to - 1],
|
||||
value: match.value,
|
||||
});
|
||||
cursor = match.to;
|
||||
}
|
||||
return matches;
|
||||
}
|
||||
|
||||
/** Whether a Lezer URL node is an email candidate rather than a written URL. */
|
||||
export function isGFMEmailURLSource(source: string): boolean {
|
||||
return source.includes("@") && !/^(?:https?:\/\/|www\.)/.test(source);
|
||||
}
|
||||
|
||||
/**
|
||||
* Find canonical GFM email ranges within Lezer textual containers. Email URL
|
||||
* nodes and backslash escapes stay in the same text run so parser splits
|
||||
* cannot shorten the decoded GFM 0.29 match; links, code, math, and other
|
||||
* syntax remain opaque.
|
||||
*/
|
||||
export function findMarkdownGFMEmailRanges(
|
||||
source: string,
|
||||
root: MarkdownSourceNode,
|
||||
resolvedLinks: MarkdownSourceRange[],
|
||||
allowUnresolvedLinks: boolean,
|
||||
): GFMEmailSourceRange[] {
|
||||
const ranges: GFMEmailSourceRange[] = [];
|
||||
|
||||
const collectUnresolvedReferenceLabel = (node: MarkdownSourceNode): void => {
|
||||
const contentFrom = node.from + 1;
|
||||
const contentTo = node.to - 1;
|
||||
if (contentFrom >= contentTo) return;
|
||||
|
||||
const prefixedSource = REFERENCE_LABEL_PREFIX + source.slice(contentFrom, contentTo);
|
||||
const root = sourceParser.parse(prefixedSource).topNode as MarkdownSourceNode;
|
||||
const offset = contentFrom - REFERENCE_LABEL_PREFIX.length;
|
||||
for (const range of findMarkdownGFMEmailRanges(prefixedSource, root, resolvedMarkdownLinkRanges(prefixedSource, root), true)) {
|
||||
const from = range.from + offset;
|
||||
const to = range.to + offset;
|
||||
if (from >= contentFrom && to <= contentTo) ranges.push({ from, to, value: range.value });
|
||||
}
|
||||
};
|
||||
|
||||
const collect = (node: MarkdownSourceNode) => {
|
||||
const linkLike = node.name === "Link" || node.name === "Image";
|
||||
const resolvedLink = linkLike && resolvedLinks.some((range) => range.from === node.from && range.to === node.to);
|
||||
if (resolvedLink || (linkLike && !allowUnresolvedLinks)) return;
|
||||
|
||||
const invalidLinkReference = node.name === "LinkReference" && !isValidMarkdownLinkReference(source, node);
|
||||
const scansDirectText = TEXT_CONTAINERS.has(node.name) || linkLike || invalidLinkReference;
|
||||
if (!scansDirectText && !STRUCTURAL_CONTAINERS.has(node.name)) return;
|
||||
|
||||
let cursor = node.from;
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) {
|
||||
const spelling = source.slice(child.from, child.to);
|
||||
const staysInTextRun =
|
||||
scansDirectText &&
|
||||
(child.name === "Escape" ||
|
||||
(child.name === "Entity" && decodeString(spelling) !== spelling) ||
|
||||
(child.name === "URL" && isGFMEmailURLSource(spelling)));
|
||||
if (staysInTextRun) continue;
|
||||
|
||||
if (scansDirectText && cursor < child.from) ranges.push(...findGFMEmailMatches(source, cursor, child.from));
|
||||
if ((linkLike || invalidLinkReference) && child.name === "LinkLabel") {
|
||||
collectUnresolvedReferenceLabel(child);
|
||||
} else {
|
||||
collect(child);
|
||||
}
|
||||
cursor = child.to;
|
||||
}
|
||||
if (scansDirectText && cursor < node.to) ranges.push(...findGFMEmailMatches(source, cursor, node.to));
|
||||
};
|
||||
|
||||
collect(root);
|
||||
return ranges.sort((left, right) => left.from - right.from || left.to - right.to);
|
||||
}
|
||||
219
web/src/utils/gfm-url.ts
Normal file
219
web/src/utils/gfm-url.ts
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
import { decodeString } from "micromark-util-decode-string";
|
||||
import type { MarkdownSourceNode } from "@/utils/gfm-email";
|
||||
import { hasExactMarkdownRange, isValidMarkdownLinkReference, type MarkdownSourceRange } from "@/utils/markdown-link";
|
||||
|
||||
export interface GFMURLSourceRange {
|
||||
from: number;
|
||||
to: number;
|
||||
}
|
||||
|
||||
const UNICODE_ALPHANUMERIC = /[\p{L}\p{N}]/u;
|
||||
const UNICODE_SEPARATOR = /\p{Z}/u;
|
||||
const TRAILING_PUNCTUATION = new Set(["!", '"', "'", "*", ",", ".", ":", ";", "?", "_", "~"]);
|
||||
|
||||
function prefixEnd(source: string, from: number): number | undefined {
|
||||
if (source.startsWith("https://", from)) return from + "https://".length;
|
||||
if (source.startsWith("http://", from)) return from + "http://".length;
|
||||
if (source.startsWith("www.", from)) return from + "www.".length;
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function domainCharacterSize(source: string, offset: number): number {
|
||||
const char = String.fromCodePoint(source.codePointAt(offset) ?? 0);
|
||||
return char === "_" || char === "-" || char === "." || UNICODE_ALPHANUMERIC.test(char) ? char.length : 0;
|
||||
}
|
||||
|
||||
function hasValidDomain(source: string, from: number, to: number): boolean {
|
||||
while (to > from && source[to - 1] === ".") to--;
|
||||
const segments = source.slice(from, to).split(".");
|
||||
return segments.length >= 2 && segments.every(Boolean) && !segments.at(-2)?.includes("_") && !segments.at(-1)?.includes("_");
|
||||
}
|
||||
|
||||
function trimGFMURLTail(source: string, from: number, to: number): number {
|
||||
const path = source.slice(from, to);
|
||||
let openParentheses = 0;
|
||||
let closeParentheses = 0;
|
||||
for (const character of path) {
|
||||
if (character === "(") openParentheses++;
|
||||
else if (character === ")") closeParentheses++;
|
||||
}
|
||||
|
||||
while (to > from) {
|
||||
const entity = source.slice(from, to).match(/&[A-Za-z]+;$/)?.[0];
|
||||
if (entity) {
|
||||
to -= entity.length;
|
||||
continue;
|
||||
}
|
||||
|
||||
const character = source[to - 1];
|
||||
if (character === ")" && closeParentheses > openParentheses) {
|
||||
closeParentheses--;
|
||||
to--;
|
||||
continue;
|
||||
}
|
||||
if (character === "]" || TRAILING_PUNCTUATION.has(character)) {
|
||||
to--;
|
||||
continue;
|
||||
}
|
||||
break;
|
||||
}
|
||||
return to;
|
||||
}
|
||||
|
||||
/** Return the complete source range of a lowercase written GFM URL at `from`. */
|
||||
export function writtenGFMURLSourceRange(source: string, from: number, upperBound = source.length): GFMURLSourceRange | undefined {
|
||||
const domainFrom = prefixEnd(source, from);
|
||||
if (domainFrom === undefined || domainFrom >= upperBound) return undefined;
|
||||
|
||||
let domainTo = domainFrom;
|
||||
while (domainTo < upperBound) {
|
||||
const size = domainCharacterSize(source, domainTo);
|
||||
if (size === 0) break;
|
||||
domainTo += size;
|
||||
}
|
||||
if (!hasValidDomain(source, domainFrom, domainTo)) return undefined;
|
||||
|
||||
let to = domainTo;
|
||||
while (to < upperBound) {
|
||||
const character = String.fromCodePoint(source.codePointAt(to) ?? 0);
|
||||
const code = source.charCodeAt(to);
|
||||
if (source[to] === "<" || (code >= 0x09 && code <= 0x0d) || UNICODE_SEPARATOR.test(character)) break;
|
||||
to += character.length;
|
||||
}
|
||||
return { from, to: trimGFMURLTail(source, from, to) };
|
||||
}
|
||||
|
||||
/** Whether a complete source spelling is a written GFM URL. */
|
||||
export function isWrittenGFMURLSource(source: string): boolean {
|
||||
return writtenGFMURLSourceRange(source, 0)?.to === source.length;
|
||||
}
|
||||
|
||||
function hasWrittenURLBoundary(source: string, from: number, contextFrom: number, allowInitialBOM: boolean): boolean {
|
||||
if (from === contextFrom) return true;
|
||||
if (allowInitialBOM && contextFrom === 0 && from === 1 && source[0] === "\uFEFF") return true;
|
||||
const previous = source[from - 1];
|
||||
const code = source.charCodeAt(from - 1);
|
||||
return (code >= 0x09 && code <= 0x0d) || previous === " " || previous === "(" || previous === "*" || previous === "_" || previous === "~";
|
||||
}
|
||||
|
||||
/** Return a written GFM URL only when it starts in a valid source context. */
|
||||
export function writtenGFMURLSourceRangeAt(
|
||||
source: string,
|
||||
from: number,
|
||||
contextFrom: number,
|
||||
upperBound = source.length,
|
||||
allowInitialBOM = true,
|
||||
): GFMURLSourceRange | undefined {
|
||||
if (!hasWrittenURLBoundary(source, from, contextFrom, allowInitialBOM)) return undefined;
|
||||
return writtenGFMURLSourceRange(source, from, upperBound);
|
||||
}
|
||||
|
||||
/** Find written GFM URLs in one already-classified literal source run. */
|
||||
export function findWrittenGFMURLRanges(
|
||||
source: string,
|
||||
lowerBound = 0,
|
||||
upperBound = source.length,
|
||||
contextFrom = lowerBound,
|
||||
opaqueRanges: GFMURLSourceRange[] = [],
|
||||
allowInitialBOM = true,
|
||||
): GFMURLSourceRange[] {
|
||||
const ranges: GFMURLSourceRange[] = [];
|
||||
let opaqueIndex = 0;
|
||||
for (let offset = contextFrom; offset < upperBound; ) {
|
||||
while (opaqueIndex < opaqueRanges.length && opaqueRanges[opaqueIndex].to <= offset) opaqueIndex++;
|
||||
const opaque = opaqueRanges[opaqueIndex];
|
||||
if (opaque && opaque.from <= offset && opaque.to > offset) {
|
||||
offset = opaque.to;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (offset >= lowerBound) {
|
||||
const range = writtenGFMURLSourceRangeAt(source, offset, contextFrom, upperBound, allowInitialBOM);
|
||||
if (range) {
|
||||
ranges.push(range);
|
||||
offset = range.to;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
offset++;
|
||||
}
|
||||
return ranges;
|
||||
}
|
||||
|
||||
const URL_SOURCE_CONTAINERS = new Set([
|
||||
"ATXHeading1",
|
||||
"ATXHeading2",
|
||||
"ATXHeading3",
|
||||
"ATXHeading4",
|
||||
"ATXHeading5",
|
||||
"ATXHeading6",
|
||||
"Paragraph",
|
||||
"SetextHeading1",
|
||||
"SetextHeading2",
|
||||
"TableCell",
|
||||
"Task",
|
||||
]);
|
||||
|
||||
const URL_PREFIX_OPAQUE_NODES = new Set([
|
||||
"Autolink",
|
||||
"BlockMath",
|
||||
"Comment",
|
||||
"Escape",
|
||||
"HTMLBlock",
|
||||
"HTMLTag",
|
||||
"InlineCode",
|
||||
"InlineMath",
|
||||
"ProcessingInstruction",
|
||||
]);
|
||||
|
||||
function opaqueURLPrefixRanges(source: string, container: MarkdownSourceNode, resolvedLinks: MarkdownSourceRange[]): GFMURLSourceRange[] {
|
||||
const ranges: GFMURLSourceRange[] = [];
|
||||
|
||||
const collect = (node: MarkdownSourceNode): void => {
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) {
|
||||
const resolvedLinkLike = (child.name === "Link" || child.name === "Image") && hasExactMarkdownRange(resolvedLinks, child);
|
||||
const knownEntity =
|
||||
child.name === "Entity" && decodeString(source.slice(child.from, child.to)) !== source.slice(child.from, child.to);
|
||||
if (resolvedLinkLike || knownEntity || URL_PREFIX_OPAQUE_NODES.has(child.name)) {
|
||||
ranges.push({ from: child.from, to: child.to });
|
||||
} else {
|
||||
collect(child);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
collect(container);
|
||||
return ranges.sort((left, right) => left.from - right.from || left.to - right.to);
|
||||
}
|
||||
|
||||
/** Find canonical written GFM URLs using Markdown syntax for bracket context. */
|
||||
export function findMarkdownGFMURLRanges(
|
||||
source: string,
|
||||
root: MarkdownSourceNode,
|
||||
resolvedLinks: MarkdownSourceRange[],
|
||||
allowInitialBOM = true,
|
||||
): GFMURLSourceRange[] {
|
||||
const ranges: GFMURLSourceRange[] = [];
|
||||
|
||||
const collect = (node: MarkdownSourceNode): void => {
|
||||
const invalidLinkReference = node.name === "LinkReference" && !isValidMarkdownLinkReference(source, node);
|
||||
if (URL_SOURCE_CONTAINERS.has(node.name) || invalidLinkReference) {
|
||||
ranges.push(
|
||||
...findWrittenGFMURLRanges(
|
||||
source,
|
||||
node.from,
|
||||
node.to,
|
||||
node.from,
|
||||
opaqueURLPrefixRanges(source, node, resolvedLinks),
|
||||
allowInitialBOM,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) collect(child);
|
||||
};
|
||||
|
||||
collect(root);
|
||||
return ranges.sort((left, right) => left.from - right.from || left.to - right.to);
|
||||
}
|
||||
81
web/src/utils/markdown-link.ts
Normal file
81
web/src/utils/markdown-link.ts
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
import { fromMarkdown } from "mdast-util-from-markdown";
|
||||
import { decodeString } from "micromark-util-decode-string";
|
||||
import { normalizeIdentifier } from "micromark-util-normalize-identifier";
|
||||
import type { MarkdownSourceNode } from "@/utils/gfm-email";
|
||||
|
||||
export interface MarkdownSourceRange {
|
||||
from: number;
|
||||
to: number;
|
||||
}
|
||||
|
||||
function childrenOf(node: MarkdownSourceNode): MarkdownSourceNode[] {
|
||||
const children: MarkdownSourceNode[] = [];
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) children.push(child);
|
||||
return children;
|
||||
}
|
||||
|
||||
function normalizeLabel(source: string): string {
|
||||
return normalizeIdentifier(decodeString(source).replaceAll("\0", "<22>"));
|
||||
}
|
||||
|
||||
function labelValue(source: string, node: MarkdownSourceNode): string {
|
||||
return normalizeLabel(source.slice(node.from + 1, node.to - 1));
|
||||
}
|
||||
|
||||
function linkTextValue(source: string, marks: MarkdownSourceNode[]): string {
|
||||
return normalizeLabel(source.slice(marks[0].to, marks[1].from));
|
||||
}
|
||||
|
||||
/** Validate a Lezer reference definition against the CommonMark parser. */
|
||||
export function isValidMarkdownLinkReference(source: string, node: MarkdownSourceNode): boolean {
|
||||
if (node.name !== "LinkReference") return false;
|
||||
return fromMarkdown(source.slice(node.from, node.to)).children.some((child) => child.type === "definition");
|
||||
}
|
||||
|
||||
/**
|
||||
* Return only links that CommonMark actually resolves. Lezer also emits Link
|
||||
* nodes for unresolved reference labels and incomplete inline-link attempts;
|
||||
* those spellings remain ordinary text to the Markdown renderer.
|
||||
*/
|
||||
export function resolvedMarkdownLinkRanges(source: string, root: MarkdownSourceNode): MarkdownSourceRange[] {
|
||||
const definitions = new Set<string>();
|
||||
|
||||
const collectDefinitions = (node: MarkdownSourceNode): void => {
|
||||
if (node.name === "LinkReference") {
|
||||
const label = childrenOf(node).find((child) => child.name === "LinkLabel");
|
||||
if (label && isValidMarkdownLinkReference(source, node)) definitions.add(labelValue(source, label));
|
||||
return;
|
||||
}
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) collectDefinitions(child);
|
||||
};
|
||||
collectDefinitions(root);
|
||||
|
||||
const ranges: MarkdownSourceRange[] = [];
|
||||
const collectLinks = (node: MarkdownSourceNode): void => {
|
||||
if (node.name === "Link" || node.name === "Image") {
|
||||
const children = childrenOf(node);
|
||||
const marks = children.filter((child) => child.name === "LinkMark");
|
||||
if (marks.length < 2) return;
|
||||
|
||||
const labelClose = marks[1].to;
|
||||
const isCompleteInlineLink = source[labelClose] === "(" && node.to > labelClose && source[node.to - 1] === ")";
|
||||
const referenceLabel = children.find((child) => child.name === "LinkLabel" && child.from >= labelClose);
|
||||
const identifier =
|
||||
referenceLabel && referenceLabel.to - referenceLabel.from > 2 ? labelValue(source, referenceLabel) : linkTextValue(source, marks);
|
||||
|
||||
if (isCompleteInlineLink || definitions.has(identifier)) {
|
||||
ranges.push({ from: node.from, to: node.to });
|
||||
return;
|
||||
}
|
||||
for (const child of children) collectLinks(child);
|
||||
return;
|
||||
}
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) collectLinks(child);
|
||||
};
|
||||
collectLinks(root);
|
||||
return ranges.sort((left, right) => left.from - right.from || left.to - right.to);
|
||||
}
|
||||
|
||||
export function hasExactMarkdownRange(ranges: MarkdownSourceRange[], node: MarkdownSourceNode): boolean {
|
||||
return ranges.some((range) => range.from === node.from && range.to === node.to);
|
||||
}
|
||||
57
web/src/utils/markdown-source-boundaries.ts
Normal file
57
web/src/utils/markdown-source-boundaries.ts
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
import { decodeString } from "micromark-util-decode-string";
|
||||
|
||||
export interface MarkdownSourceBoundary {
|
||||
from: number;
|
||||
to: number;
|
||||
type: "Entity" | "Escape";
|
||||
}
|
||||
|
||||
export interface DecodedMarkdownCharacterReference {
|
||||
to: number;
|
||||
value: string;
|
||||
}
|
||||
|
||||
const CHARACTER_REFERENCE_AT_START = /^&(?:#[0-9]{1,7}|#[xX][0-9A-Fa-f]{1,6}|[A-Za-z0-9]{1,31});/;
|
||||
|
||||
function isASCIIPunctuation(character: string): boolean {
|
||||
const code = character.charCodeAt(0);
|
||||
return (
|
||||
(code >= 0x21 && code <= 0x2f) || (code >= 0x3a && code <= 0x40) || (code >= 0x5b && code <= 0x60) || (code >= 0x7b && code <= 0x7e)
|
||||
);
|
||||
}
|
||||
|
||||
/** Decode one complete CommonMark character reference starting at `from`. */
|
||||
export function decodedMarkdownCharacterReferenceAt(
|
||||
source: string,
|
||||
from: number,
|
||||
to = source.length,
|
||||
): DecodedMarkdownCharacterReference | undefined {
|
||||
if (source[from] !== "&") return undefined;
|
||||
const spelling = source.slice(from, Math.min(to, from + 35)).match(CHARACTER_REFERENCE_AT_START)?.[0];
|
||||
if (!spelling) return undefined;
|
||||
const value = decodeString(spelling);
|
||||
return value === spelling ? undefined : { to: from + spelling.length, value };
|
||||
}
|
||||
|
||||
/** Find source spellings that Markdown decodes and therefore cannot join tags. */
|
||||
export function findDecodedMarkdownSourceBoundaries(source: string, from = 0, to = source.length): MarkdownSourceBoundary[] {
|
||||
const ranges: MarkdownSourceBoundary[] = [];
|
||||
for (let cursor = from; cursor < to; ) {
|
||||
if (source[cursor] === "\\" && cursor + 1 < to && isASCIIPunctuation(source[cursor + 1])) {
|
||||
ranges.push({ from: cursor, to: cursor + 2, type: "Escape" });
|
||||
cursor += 2;
|
||||
continue;
|
||||
}
|
||||
|
||||
const reference = decodedMarkdownCharacterReferenceAt(source, cursor, to);
|
||||
if (reference) {
|
||||
ranges.push({ from: cursor, to: reference.to, type: "Entity" });
|
||||
cursor = reference.to;
|
||||
continue;
|
||||
}
|
||||
|
||||
const codePoint = source.codePointAt(cursor);
|
||||
cursor += codePoint !== undefined && codePoint > 0xffff ? 2 : 1;
|
||||
}
|
||||
return ranges;
|
||||
}
|
||||
81
web/src/utils/memo-markdown-extension.ts
Normal file
81
web/src/utils/memo-markdown-extension.ts
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
import { GFM, type MarkdownConfig, type MarkdownExtension } from "@lezer/markdown";
|
||||
|
||||
function openingMathFenceSize(line: { text: string; pos: number }): number | undefined {
|
||||
let end = line.pos;
|
||||
while (line.text.charCodeAt(end) === 0x24) end++;
|
||||
const size = end - line.pos;
|
||||
if (size < 2 || line.text.indexOf("$", end) >= 0) return undefined;
|
||||
return size;
|
||||
}
|
||||
|
||||
function isClosingMathFence(line: { text: string; pos: number; skipSpace(from: number): number }, openingSize: number): boolean {
|
||||
let end = line.pos;
|
||||
while (line.text.charCodeAt(end) === 0x24) end++;
|
||||
return end - line.pos >= openingSize && line.skipSpace(end) === line.text.length;
|
||||
}
|
||||
|
||||
const mathExtension: MarkdownConfig = {
|
||||
defineNodes: ["InlineMath", "MathMark", { name: "BlockMath", block: true }],
|
||||
parseBlock: [
|
||||
{
|
||||
name: "BlockMath",
|
||||
parse(context, line) {
|
||||
const openingSize = openingMathFenceSize(line);
|
||||
if (!openingSize) return false;
|
||||
|
||||
const from = context.lineStart + line.pos;
|
||||
// This is the same scope guard used by Lezer's fenced-code parser. It
|
||||
// keeps an unclosed math block inside its current list or blockquote.
|
||||
const activeDepth = context.depth;
|
||||
while (context.nextLine() && (line as typeof line & { depth: number }).depth >= activeDepth) {
|
||||
if (isClosingMathFence(line, openingSize)) {
|
||||
context.nextLine();
|
||||
break;
|
||||
}
|
||||
}
|
||||
context.addElement(context.elt("BlockMath", from, context.prevLineEnd()));
|
||||
return true;
|
||||
},
|
||||
endLeaf(_context, line) {
|
||||
return openingMathFenceSize(line) !== undefined;
|
||||
},
|
||||
before: "FencedCode",
|
||||
},
|
||||
],
|
||||
parseInline: [
|
||||
{
|
||||
name: "InlineMath",
|
||||
parse(context, next, position) {
|
||||
if (next !== 0x24) return -1;
|
||||
|
||||
let openingEnd = position + 1;
|
||||
while (context.char(openingEnd) === 0x24) openingEnd++;
|
||||
const openingSize = openingEnd - position;
|
||||
for (let cursor = openingEnd; cursor < context.end; ) {
|
||||
if (context.char(cursor) !== 0x24) {
|
||||
cursor++;
|
||||
continue;
|
||||
}
|
||||
let closingEnd = cursor + 1;
|
||||
while (context.char(closingEnd) === 0x24) closingEnd++;
|
||||
if (closingEnd - cursor === openingSize) {
|
||||
return context.addElement(
|
||||
context.elt("InlineMath", position, closingEnd, [
|
||||
context.elt("MathMark", position, openingEnd),
|
||||
context.elt("MathMark", cursor, closingEnd),
|
||||
]),
|
||||
);
|
||||
}
|
||||
cursor = closingEnd;
|
||||
}
|
||||
// An unmatched opener is ordinary text. Consume the complete run so
|
||||
// a later dollar cannot reinterpret its tail as a shorter opener.
|
||||
return openingEnd;
|
||||
},
|
||||
after: "Escape",
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
/** GFM plus the opaque dollar-math nodes understood by the memo renderer. */
|
||||
export const memoMarkdownExtensions: MarkdownExtension = [GFM, mathExtension];
|
||||
|
|
@ -11,13 +11,22 @@ function isMentionBoundary(char: string): boolean {
|
|||
|
||||
type Segment = { type: "text"; value: string } | { type: "mention"; value: string };
|
||||
|
||||
export function parseMentionsFromText(text: string): Segment[] {
|
||||
export function parseMentionsFromText(text: string, leadingChar = ""): Segment[] {
|
||||
const segments: Segment[] = [];
|
||||
const chars = [...text];
|
||||
let i = 0;
|
||||
|
||||
const appendText = (value: string) => {
|
||||
const previous = segments.at(-1);
|
||||
if (previous?.type === "text") {
|
||||
previous.value += value;
|
||||
} else {
|
||||
segments.push({ type: "text", value });
|
||||
}
|
||||
};
|
||||
|
||||
while (i < chars.length) {
|
||||
const prevChar = i > 0 ? chars[i - 1] : "";
|
||||
const prevChar = i > 0 ? chars[i - 1] : leadingChar;
|
||||
if (chars[i] === "@" && isMentionBoundary(prevChar) && i + 1 < chars.length && isMentionChar(chars[i + 1])) {
|
||||
let j = i + 1;
|
||||
while (j < chars.length && isMentionChar(chars[j]) && j - i - 1 < MAX_MENTION_LENGTH) {
|
||||
|
|
@ -38,7 +47,7 @@ export function parseMentionsFromText(text: string): Segment[] {
|
|||
while (j < chars.length && chars[j] !== "@") {
|
||||
j++;
|
||||
}
|
||||
segments.push({ type: "text", value: chars.slice(i, j).join("") });
|
||||
appendText(chars.slice(i, j).join(""));
|
||||
i = j;
|
||||
}
|
||||
|
||||
|
|
@ -69,13 +78,63 @@ function createMentionNode(username: string): MentionNode {
|
|||
} as MentionNode;
|
||||
}
|
||||
|
||||
function trailingText(node: UnistNode | undefined): string {
|
||||
if (!node) return "";
|
||||
if (node.type === "text") return (node as Text).value;
|
||||
const nodeChildren = (node as { children?: UnistNode[] }).children;
|
||||
if (nodeChildren?.length) return trailingText(nodeChildren.at(-1));
|
||||
const children = (node.data as { hChildren?: Array<{ type?: string; value?: string }> } | undefined)?.hChildren;
|
||||
const last = children?.at(-1);
|
||||
return last?.type === "text" ? (last.value ?? "") : "";
|
||||
}
|
||||
|
||||
function previousSourceCharacter(source: string, offset: number): string {
|
||||
return [...source.slice(Math.max(0, offset - 2), offset)].at(-1) ?? "";
|
||||
}
|
||||
|
||||
function synthesizedLiteralTrailingText(node: UnistNode | undefined, source: string): string {
|
||||
if (!node) return "";
|
||||
if (node.type === "tagNode" || node.type === "mentionNode") return trailingText(node);
|
||||
if (node.type !== "link" || !(node as { url?: string }).url?.startsWith("mailto:")) return "";
|
||||
|
||||
const rendered = trailingText(node);
|
||||
const from = node.position?.start.offset;
|
||||
const to = node.position?.end.offset;
|
||||
if (from !== undefined && to !== undefined && source.slice(from, to) !== rendered) {
|
||||
// Explicit links and autolinks end in Markdown syntax (`)` or `>`), which
|
||||
// is the mention boundary. Only a literal GFM email contributes its final
|
||||
// rendered character across an mdast child boundary.
|
||||
return "";
|
||||
}
|
||||
return rendered;
|
||||
}
|
||||
|
||||
function leadingCharacter(parent: UnistNode, index: number, node: Text, source: string): string {
|
||||
const previous = (parent as { children?: UnistNode[] }).children?.[index - 1];
|
||||
if (!previous) return "";
|
||||
|
||||
const currentFrom = node.position?.start.offset;
|
||||
const previousTo = previous.position?.end.offset;
|
||||
if (currentFrom !== undefined && previousTo === currentFrom) {
|
||||
// Use the actual source boundary instead of text rendered by an arbitrary
|
||||
// preceding node. Formatting and link delimiters are meaningful mention
|
||||
// boundaries even though they are not present in rendered text.
|
||||
return previousSourceCharacter(source, currentFrom);
|
||||
}
|
||||
|
||||
return [...synthesizedLiteralTrailingText(previous, source)].at(-1) ?? "";
|
||||
}
|
||||
|
||||
type VFileLike = { value?: string | Uint8Array };
|
||||
|
||||
export const remarkMention = () => {
|
||||
return (tree: Root) => {
|
||||
return (tree: Root, file: VFileLike) => {
|
||||
const source = typeof file.value === "string" ? file.value : "";
|
||||
visit(tree, (node, index, parent) => {
|
||||
if (node.type !== "text" || !parent || index === null) return;
|
||||
if (node.type !== "text" || !parent || parent.type === "link" || typeof index !== "number") return;
|
||||
|
||||
const textNode = node as Text;
|
||||
const segments = parseMentionsFromText(textNode.value);
|
||||
const segments = parseMentionsFromText(textNode.value, leadingCharacter(parent, index, textNode, source));
|
||||
if (segments.every((segment) => segment.type === "text")) {
|
||||
return;
|
||||
}
|
||||
|
|
@ -90,9 +149,7 @@ export const remarkMention = () => {
|
|||
} as Text;
|
||||
});
|
||||
|
||||
if (typeof index === "number") {
|
||||
(parent.children as UnistNode[]).splice(index, 1, ...(newNodes as UnistNode[]));
|
||||
}
|
||||
(parent.children as UnistNode[]).splice(index, 1, ...(newNodes as UnistNode[]));
|
||||
});
|
||||
};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,134 +1,370 @@
|
|||
import type { Root, Text } from "mdast";
|
||||
import type { Position, Node as UnistNode } from "unist";
|
||||
import { parser as markdownParser } from "@lezer/markdown";
|
||||
import type { Link, Root, Text } from "mdast";
|
||||
import { decodeString } from "micromark-util-decode-string";
|
||||
import type { Node as UnistNode } from "unist";
|
||||
import type { TagNode, TagNodeData } from "@/types/markdown";
|
||||
import { isTagChar, MAX_TAG_LENGTH } from "@/utils/tag-grammar";
|
||||
import { findMarkdownGFMEmailRanges, type GFMEmailSourceRange, type MarkdownSourceNode } from "@/utils/gfm-email";
|
||||
import { findMarkdownGFMURLRanges } from "@/utils/gfm-url";
|
||||
import { hasExactMarkdownRange, resolvedMarkdownLinkRanges } from "@/utils/markdown-link";
|
||||
import { decodedMarkdownCharacterReferenceAt, findDecodedMarkdownSourceBoundaries } from "@/utils/markdown-source-boundaries";
|
||||
import { memoMarkdownExtensions } from "@/utils/memo-markdown-extension";
|
||||
import { findTagMatches } from "@/utils/tag-grammar";
|
||||
|
||||
type Segment = { type: "text"; value: string } | { type: "tag"; value: string };
|
||||
type Segment = { type: "text"; value: string } | { type: "tag"; source: string; value: string };
|
||||
|
||||
// CommonMark "ASCII punctuation": the only characters a leading backslash can
|
||||
// escape. A backslash before anything else is a literal backslash.
|
||||
function isAsciiPunctuation(char: string): boolean {
|
||||
if (char.length !== 1) {
|
||||
return false;
|
||||
}
|
||||
const code = char.charCodeAt(0);
|
||||
return (
|
||||
(code >= 0x21 && code <= 0x2f) || // ! " # $ % & ' ( ) * + , - . /
|
||||
(code >= 0x3a && code <= 0x40) || // : ; < = > ? @
|
||||
(code >= 0x5b && code <= 0x60) || // [ \ ] ^ _ `
|
||||
(code >= 0x7b && code <= 0x7e) // { | } ~
|
||||
interface SourceRange {
|
||||
from: number;
|
||||
to: number;
|
||||
type: "Entity" | "Escape" | "LineEnding" | "Opaque";
|
||||
}
|
||||
|
||||
interface MarkdownSourceContext {
|
||||
boundaries: SourceRange[];
|
||||
emailRanges: GFMEmailSourceRange[];
|
||||
emphasisRanges: Array<{ from: number; to: number; delimiterWidth: 1 | 2 }>;
|
||||
linkRanges: Array<{ from: number; to: number }>;
|
||||
syntaxLinkRanges: Array<{ from: number; to: number }>;
|
||||
}
|
||||
|
||||
interface LiteralRun {
|
||||
source: string;
|
||||
sourceFrom: number;
|
||||
valueFrom: number;
|
||||
}
|
||||
|
||||
interface SourceSpan {
|
||||
from: number;
|
||||
to: number;
|
||||
}
|
||||
|
||||
const sourceParser = markdownParser.configure(memoMarkdownExtensions);
|
||||
const TRANSPARENT_PARENT_TYPES = new Set([
|
||||
"blockquote",
|
||||
"delete",
|
||||
"emphasis",
|
||||
"heading",
|
||||
"list",
|
||||
"listItem",
|
||||
"paragraph",
|
||||
"root",
|
||||
"strong",
|
||||
"table",
|
||||
"tableCell",
|
||||
"tableRow",
|
||||
]);
|
||||
|
||||
function distinctRanges<T extends SourceSpan>(values: T[], discriminator: (value: T) => string = () => ""): T[] {
|
||||
const seen = new Set<string>();
|
||||
return values.filter((value) => {
|
||||
const key = `${value.from}:${value.to}:${discriminator(value)}`;
|
||||
if (seen.has(key)) return false;
|
||||
seen.add(key);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
function markdownSourceContext(source: string): MarkdownSourceContext {
|
||||
const ranges: SourceRange[] = [];
|
||||
const emphasisRanges: Array<{ from: number; to: number; delimiterWidth: 1 | 2 }> = [];
|
||||
const linkRanges: Array<{ from: number; to: number }> = [];
|
||||
const syntaxLinkRanges: Array<{ from: number; to: number }> = [];
|
||||
const tree = sourceParser.parse(source);
|
||||
const sourceRoot = tree.topNode as MarkdownSourceNode;
|
||||
const resolvedLinks = resolvedMarkdownLinkRanges(source, sourceRoot);
|
||||
const emailRanges = findMarkdownGFMEmailRanges(source, sourceRoot, resolvedLinks, true);
|
||||
// Micromark has already consumed the one permitted document BOM before the
|
||||
// source reaches this reconciler, so a remaining BOM is ordinary text.
|
||||
const writtenURLRanges = findMarkdownGFMURLRanges(source, sourceRoot, resolvedLinks, false);
|
||||
linkRanges.push(...writtenURLRanges);
|
||||
ranges.push(...writtenURLRanges.map((range) => ({ ...range, type: "Opaque" as const })));
|
||||
const decodedBoundaries = findDecodedMarkdownSourceBoundaries(source);
|
||||
tree.iterate({
|
||||
enter(node) {
|
||||
if (node.name === "Escape" || node.name === "Entity") {
|
||||
ranges.push({ from: node.from, to: node.to, type: node.name });
|
||||
return false;
|
||||
}
|
||||
if (node.name === "InlineMath" || node.name === "BlockMath") {
|
||||
ranges.push({ from: node.from, to: node.to, type: "Opaque" });
|
||||
return false;
|
||||
}
|
||||
if (node.name === "InlineCode") {
|
||||
ranges.push({ from: node.from, to: node.to, type: "Opaque" });
|
||||
return false;
|
||||
}
|
||||
if (node.name === "EmphasisMark") {
|
||||
ranges.push({ from: node.from, to: node.to, type: "Opaque" });
|
||||
return false;
|
||||
}
|
||||
if (node.name === "Emphasis" || node.name === "StrongEmphasis") {
|
||||
emphasisRanges.push({ from: node.from, to: node.to, delimiterWidth: node.name === "StrongEmphasis" ? 2 : 1 });
|
||||
}
|
||||
if (node.name === "Link" || node.name === "Image" || node.name === "Autolink") {
|
||||
if (node.name !== "Autolink" && !hasExactMarkdownRange(resolvedLinks, node.node as MarkdownSourceNode)) return;
|
||||
linkRanges.push({ from: node.from, to: node.to });
|
||||
syntaxLinkRanges.push({ from: node.from, to: node.to });
|
||||
ranges.push({ from: node.from, to: node.to, type: "Opaque" });
|
||||
return false;
|
||||
}
|
||||
},
|
||||
});
|
||||
ranges.push(
|
||||
...decodedBoundaries.filter(
|
||||
(boundary) => !ranges.some((range) => range.type === "Opaque" && range.from <= boundary.from && range.to >= boundary.to),
|
||||
),
|
||||
);
|
||||
|
||||
const projectUnresolvedSecondLabels = (node: MarkdownSourceNode): void => {
|
||||
const unresolvedLinkLike = (node.name === "Link" || node.name === "Image") && !hasExactMarkdownRange(resolvedLinks, node);
|
||||
if (unresolvedLinkLike) {
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) {
|
||||
if (child.name !== "LinkLabel" || child.to - child.from <= 2) continue;
|
||||
|
||||
const contentFrom = child.from + 1;
|
||||
const contentTo = child.to - 1;
|
||||
const prefix = "x ";
|
||||
const nested = markdownSourceContext(prefix + source.slice(contentFrom, contentTo));
|
||||
const offset = contentFrom - prefix.length;
|
||||
const project = <T extends { from: number; to: number }>(ranges: T[]): T[] =>
|
||||
ranges
|
||||
.filter((range) => range.from >= prefix.length && range.to <= prefix.length + contentTo - contentFrom)
|
||||
.map((range) => ({ ...range, from: range.from + offset, to: range.to + offset }));
|
||||
|
||||
const excludedSyntheticBoundaryURLs = nested.linkRanges.filter(
|
||||
(range) =>
|
||||
range.from === prefix.length &&
|
||||
!nested.syntaxLinkRanges.some((syntaxRange) => syntaxRange.from === range.from && syntaxRange.to === range.to),
|
||||
);
|
||||
const isExcludedSyntheticBoundaryURL = (range: { from: number; to: number }) =>
|
||||
excludedSyntheticBoundaryURLs.some((excluded) => excluded.from === range.from && excluded.to === range.to);
|
||||
|
||||
ranges.push(...project(nested.boundaries.filter((range) => !isExcludedSyntheticBoundaryURL(range))));
|
||||
emphasisRanges.push(...project(nested.emphasisRanges));
|
||||
linkRanges.push(...project(nested.linkRanges.filter((range) => !isExcludedSyntheticBoundaryURL(range))));
|
||||
syntaxLinkRanges.push(...project(nested.syntaxLinkRanges));
|
||||
}
|
||||
}
|
||||
for (let child = node.firstChild; child; child = child.nextSibling) projectUnresolvedSecondLabels(child);
|
||||
};
|
||||
projectUnresolvedSecondLabels(sourceRoot);
|
||||
|
||||
const projectRejectedURLNodes = (parent: MarkdownSourceNode): void => {
|
||||
if (
|
||||
parent.name === "Autolink" ||
|
||||
parent.name === "LinkReference" ||
|
||||
((parent.name === "Link" || parent.name === "Image") && hasExactMarkdownRange(resolvedLinks, parent))
|
||||
) {
|
||||
return;
|
||||
}
|
||||
for (let node = parent.firstChild; node; node = node.nextSibling) {
|
||||
if (node.name !== "URL") {
|
||||
projectRejectedURLNodes(node);
|
||||
continue;
|
||||
}
|
||||
|
||||
const coveredByCanonicalURL = writtenURLRanges.some((range) => range.from <= node.from && range.to >= node.to);
|
||||
const coveredByEmail = emailRanges.some((range) => range.from <= node.from && range.to >= node.to);
|
||||
if (!coveredByCanonicalURL && !coveredByEmail) {
|
||||
const prefix = "x";
|
||||
const nestedSource = prefix + source.slice(node.from, node.to);
|
||||
const nested = markdownSourceContext(nestedSource);
|
||||
const offset = node.from - prefix.length;
|
||||
const project = <T extends { from: number; to: number }>(values: T[]): T[] =>
|
||||
values
|
||||
.filter((range) => range.from >= prefix.length && range.to <= nestedSource.length)
|
||||
.map((range) => ({ ...range, from: range.from + offset, to: range.to + offset }));
|
||||
|
||||
ranges.push(...project(nested.boundaries));
|
||||
emailRanges.push(...project(nested.emailRanges));
|
||||
emphasisRanges.push(...project(nested.emphasisRanges));
|
||||
linkRanges.push(...project(nested.linkRanges));
|
||||
syntaxLinkRanges.push(...project(nested.syntaxLinkRanges));
|
||||
|
||||
if (source.slice(node.from, node.to).includes("_") && source.slice(parent.from, node.from).includes("_")) {
|
||||
const insertion = node.from - parent.from;
|
||||
const contextualSource = source.slice(parent.from, node.from) + prefix + source.slice(node.from, node.to);
|
||||
const contentFrom = insertion + prefix.length;
|
||||
sourceParser.parse(contextualSource).iterate({
|
||||
enter(mark) {
|
||||
if (mark.name === "EmphasisMark" && mark.from >= contentFrom) {
|
||||
ranges.push({
|
||||
from: mark.from + parent.from - prefix.length,
|
||||
to: mark.to + parent.from - prefix.length,
|
||||
type: "Opaque",
|
||||
});
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
projectRejectedURLNodes(sourceRoot);
|
||||
|
||||
for (const match of source.matchAll(/\r\n?|\n/g)) {
|
||||
const lineEndingTo = match.index + match[0].length;
|
||||
if (!ranges.some((range) => range.type === "Opaque" && range.from < lineEndingTo && range.to > match.index)) {
|
||||
ranges.push({ from: match.index, to: lineEndingTo, type: "LineEnding" });
|
||||
}
|
||||
}
|
||||
return {
|
||||
boundaries: distinctRanges(ranges, (range) => range.type).sort((left, right) => left.from - right.from || left.to - right.to),
|
||||
emailRanges: distinctRanges(emailRanges, (range) => range.value),
|
||||
emphasisRanges: distinctRanges(emphasisRanges, (range) => String(range.delimiterWidth)).sort(
|
||||
(left, right) => left.to - left.from - (right.to - right.from) || left.from - right.from,
|
||||
),
|
||||
linkRanges: distinctRanges(linkRanges).sort((left, right) => left.from - right.from || left.to - right.to),
|
||||
syntaxLinkRanges: distinctRanges(syntaxLinkRanges).sort((left, right) => left.from - right.from || left.to - right.to),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply CommonMark backslash-unescaping to a raw source slice, tracking which
|
||||
* resulting characters came from an escape. `\#` yields a `#` flagged escaped,
|
||||
* so the tag lexer can tell a deliberately-escaped hash from a real tag.
|
||||
* Returns code points (not UTF-16 units) so astral characters stay intact.
|
||||
* Map original-source literal runs to the decoded mdast text value. Markdown
|
||||
* escapes and character references are omitted from the runs and therefore
|
||||
* remain hard lexer boundaries.
|
||||
*/
|
||||
function unescapeBackslashes(source: string): { chars: string[]; escaped: boolean[] } {
|
||||
const codePoints = [...source];
|
||||
const chars: string[] = [];
|
||||
const escaped: boolean[] = [];
|
||||
function literalRunsForTextNode(
|
||||
value: string,
|
||||
span: SourceSpan | undefined,
|
||||
source: string,
|
||||
boundaries: SourceRange[],
|
||||
): LiteralRun[] | undefined {
|
||||
if (!span) return undefined;
|
||||
const { from: start, to: end } = span;
|
||||
|
||||
for (let i = 0; i < codePoints.length; i++) {
|
||||
if (codePoints[i] === "\\" && i + 1 < codePoints.length && isAsciiPunctuation(codePoints[i + 1])) {
|
||||
chars.push(codePoints[i + 1]);
|
||||
escaped.push(true);
|
||||
i++;
|
||||
const runs: LiteralRun[] = [];
|
||||
let sourceCursor = start;
|
||||
let valueCursor = 0;
|
||||
let maySkipContainerPrefix = false;
|
||||
|
||||
const decodeBoundary = (spelling: string, type: SourceRange["type"]): string => {
|
||||
if (type !== "LineEnding") return decodeString(spelling).replaceAll("\0", "<22>");
|
||||
return /^(?:\r\n?|\n)/.test(spelling) ? "\n" : "";
|
||||
};
|
||||
|
||||
const addRun = (runEnd: number, mayOmitTrailingWhitespace = false): boolean => {
|
||||
let runStart = sourceCursor;
|
||||
let runSource = source.slice(runStart, runEnd);
|
||||
if (!runSource) return true;
|
||||
|
||||
let decodedRun = runSource.replaceAll("\0", "<22>");
|
||||
if (!value.startsWith(decodedRun, valueCursor)) {
|
||||
if (mayOmitTrailingWhitespace) {
|
||||
let trimmedEnd = runEnd;
|
||||
while (trimmedEnd > runStart && (source[trimmedEnd - 1] === " " || source[trimmedEnd - 1] === "\t")) trimmedEnd--;
|
||||
runSource = source.slice(runStart, trimmedEnd);
|
||||
decodedRun = runSource.replaceAll("\0", "<22>");
|
||||
if (value.startsWith(decodedRun, valueCursor)) {
|
||||
if (runSource) runs.push({ source: runSource, sourceFrom: runStart, valueFrom: valueCursor });
|
||||
valueCursor += decodedRun.length;
|
||||
maySkipContainerPrefix = false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
if (!maySkipContainerPrefix) return false;
|
||||
let aligned = false;
|
||||
while (runStart < runEnd && (source[runStart] === " " || source[runStart] === "\t" || source[runStart] === ">")) {
|
||||
runStart++;
|
||||
runSource = source.slice(runStart, runEnd);
|
||||
decodedRun = runSource.replaceAll("\0", "<22>");
|
||||
if (value.startsWith(decodedRun, valueCursor)) {
|
||||
aligned = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!aligned) return false;
|
||||
}
|
||||
|
||||
if (runSource) runs.push({ source: runSource, sourceFrom: runStart, valueFrom: valueCursor });
|
||||
valueCursor += decodedRun.length;
|
||||
maySkipContainerPrefix = false;
|
||||
return true;
|
||||
};
|
||||
|
||||
for (const boundary of boundaries) {
|
||||
const boundaryFrom = Math.max(boundary.from, start);
|
||||
const boundaryTo = Math.min(boundary.to, end);
|
||||
if (boundaryFrom >= boundaryTo) continue;
|
||||
|
||||
const fullBoundarySource = source.slice(boundary.from, boundary.to);
|
||||
const fullDecodedBoundary = decodeBoundary(fullBoundarySource, boundary.type);
|
||||
if (boundary.type === "Entity" && fullDecodedBoundary === fullBoundarySource) {
|
||||
// Lezer recognizes the shape of unknown named references. Micromark
|
||||
// leaves those names literal, so they are not syntax boundaries.
|
||||
continue;
|
||||
}
|
||||
chars.push(codePoints[i]);
|
||||
escaped.push(false);
|
||||
|
||||
const boundarySource = source.slice(boundaryFrom, boundaryTo);
|
||||
const decodedBoundary = decodeBoundary(boundarySource, boundary.type);
|
||||
|
||||
if (!addRun(boundaryFrom, boundary.type === "LineEnding")) return undefined;
|
||||
sourceCursor = boundaryTo;
|
||||
if (boundary.type === "LineEnding") {
|
||||
const rawLineEnding = boundarySource.match(/^(?:\r\n?|\n)/)?.[0] ?? "";
|
||||
if (rawLineEnding && value.startsWith(rawLineEnding, valueCursor)) {
|
||||
valueCursor += rawLineEnding.length;
|
||||
} else if (rawLineEnding && value.startsWith("\n", valueCursor)) {
|
||||
valueCursor++;
|
||||
} else if (rawLineEnding) {
|
||||
return undefined;
|
||||
}
|
||||
maySkipContainerPrefix = true;
|
||||
continue;
|
||||
}
|
||||
// remark-gfm sometimes preserves escape/entity source inside a permissive
|
||||
// literal link. Prefer that complete raw spelling: a decoded entity such
|
||||
// as `&` can otherwise be a misleading prefix of `&`.
|
||||
const rawBoundary = boundarySource.replaceAll("\0", "<22>");
|
||||
if (value.startsWith(rawBoundary, valueCursor)) {
|
||||
valueCursor += rawBoundary.length;
|
||||
} else {
|
||||
if (!value.startsWith(decodedBoundary, valueCursor)) return undefined;
|
||||
valueCursor += decodedBoundary.length;
|
||||
}
|
||||
}
|
||||
|
||||
return { chars, escaped };
|
||||
if (!addRun(end)) return undefined;
|
||||
return runs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Split a run of characters into text/tag segments. `escaped[i]` marks a
|
||||
* character that came from a backslash escape: an escaped `#` can never start a
|
||||
* tag (so `\#NAS` stays literal text), matching the backend goldmark parser and
|
||||
* the editor's marked tokenizer, both of which honor the escape natively.
|
||||
*/
|
||||
function parseSegments(chars: string[], escaped: boolean[]): Segment[] {
|
||||
function segmentsForTextNode(
|
||||
value: string,
|
||||
span: SourceSpan | undefined,
|
||||
source: string,
|
||||
boundaries: SourceRange[],
|
||||
): Segment[] | undefined {
|
||||
const runs = literalRunsForTextNode(value, span, source, boundaries);
|
||||
if (!runs) return undefined;
|
||||
|
||||
const tags = runs.flatMap((run) =>
|
||||
findTagMatches(run.source).map((match) => ({
|
||||
from: run.valueFrom + match.from,
|
||||
to: run.valueFrom + match.to,
|
||||
source: run.source.slice(match.from, match.to),
|
||||
value: match.value,
|
||||
})),
|
||||
);
|
||||
if (tags.length === 0) return [{ type: "text", value }];
|
||||
|
||||
const segments: Segment[] = [];
|
||||
let i = 0;
|
||||
|
||||
while (i < chars.length) {
|
||||
if (chars[i] === "#" && !escaped[i] && i + 1 < chars.length && isTagChar(chars[i + 1])) {
|
||||
const prevChar = i > 0 ? chars[i - 1] : "";
|
||||
const nextChar = chars[i + 1];
|
||||
|
||||
if (prevChar === "#" || nextChar === "#" || nextChar === " ") {
|
||||
segments.push({ type: "text", value: chars[i] });
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let j = i + 1;
|
||||
while (j < chars.length && isTagChar(chars[j])) {
|
||||
j++;
|
||||
}
|
||||
|
||||
const tagContent = chars.slice(i + 1, j).join("");
|
||||
|
||||
const runeCount = [...tagContent].length;
|
||||
if (runeCount > 0 && runeCount <= MAX_TAG_LENGTH) {
|
||||
segments.push({ type: "tag", value: tagContent });
|
||||
i = j;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// Consume a plain-text run up to the next tag-eligible (non-escaped) hash.
|
||||
let j = i + 1;
|
||||
while (j < chars.length && !(chars[j] === "#" && !escaped[j])) {
|
||||
j++;
|
||||
}
|
||||
segments.push({ type: "text", value: chars.slice(i, j).join("") });
|
||||
i = j;
|
||||
let cursor = 0;
|
||||
for (const tag of tags) {
|
||||
if (cursor < tag.from) segments.push({ type: "text", value: value.slice(cursor, tag.from) });
|
||||
segments.push({ type: "tag", source: tag.source, value: tag.value });
|
||||
cursor = tag.to;
|
||||
}
|
||||
|
||||
if (cursor < value.length) segments.push({ type: "text", value: value.slice(cursor) });
|
||||
return segments;
|
||||
}
|
||||
|
||||
/**
|
||||
* Segment a text node, preferring the original source slice (where escapes are
|
||||
* still visible) over the post-escape `value`. The source-derived parse is only
|
||||
* trusted when it reconstructs `value` byte-for-byte; otherwise — entity
|
||||
* references, missing positions, upstream rewrites — we fall back to the
|
||||
* value-based parse, which is exactly the pre-escape-support behavior. This
|
||||
* makes escape support strictly additive: it never changes a node it can't
|
||||
* faithfully account for.
|
||||
*/
|
||||
function segmentsForTextNode(value: string, position: Position | undefined, source: string): Segment[] {
|
||||
const startOffset = position?.start?.offset;
|
||||
const endOffset = position?.end?.offset;
|
||||
|
||||
if (source && startOffset != null && endOffset != null) {
|
||||
const slice = source.slice(startOffset, endOffset);
|
||||
const { chars, escaped } = unescapeBackslashes(slice);
|
||||
if (chars.join("") === value) {
|
||||
return parseSegments(chars, escaped);
|
||||
}
|
||||
}
|
||||
|
||||
const chars = [...value];
|
||||
return parseSegments(
|
||||
chars,
|
||||
chars.map(() => false),
|
||||
);
|
||||
}
|
||||
|
||||
function createTagNode(tagValue: string): TagNode {
|
||||
function createTagNode(tagValue: string, source: string): TagNode {
|
||||
const data: TagNodeData = {
|
||||
hName: "span",
|
||||
hProperties: {
|
||||
className: "tag",
|
||||
"data-tag": tagValue,
|
||||
},
|
||||
hChildren: [{ type: "text", value: `#${tagValue}` }],
|
||||
hChildren: [{ type: "text", value: source }],
|
||||
};
|
||||
|
||||
return {
|
||||
|
|
@ -144,40 +380,330 @@ function isParentNode(node: UnistNode): node is ParentNode {
|
|||
return Array.isArray((node as { children?: unknown }).children);
|
||||
}
|
||||
|
||||
function isLinkNode(node: UnistNode): boolean {
|
||||
return node.type === "link" || node.type === "linkReference";
|
||||
function positionedSpan(node: UnistNode): SourceSpan | undefined {
|
||||
const from = node.position?.start.offset;
|
||||
const to = node.position?.end.offset;
|
||||
return from === undefined || to === undefined ? undefined : { from, to };
|
||||
}
|
||||
|
||||
function transformTagTextNodes(parent: ParentNode, insideLink: boolean, source: string): void {
|
||||
for (let index = 0; index < parent.children.length; index++) {
|
||||
const child = parent.children[index];
|
||||
const childInsideLink = insideLink || isLinkNode(child);
|
||||
function nodeSpan(node: UnistNode, recoveredSpans: WeakMap<UnistNode, SourceSpan>): SourceSpan | undefined {
|
||||
return positionedSpan(node) ?? recoveredSpans.get(node);
|
||||
}
|
||||
|
||||
if (child.type === "text" && !childInsideLink) {
|
||||
const textNode = child as Text;
|
||||
const segments = segmentsForTextNode(textNode.value, textNode.position, source);
|
||||
function isCoveredBySourceRange(
|
||||
node: UnistNode,
|
||||
ranges: Array<{ from: number; to: number }>,
|
||||
recoveredSpans: WeakMap<UnistNode, SourceSpan>,
|
||||
): boolean {
|
||||
const span = nodeSpan(node, recoveredSpans);
|
||||
return !!span && ranges.some((range) => range.from <= span.from && range.to >= span.to);
|
||||
}
|
||||
|
||||
if (segments.every((seg) => seg.type === "text")) {
|
||||
function isWrittenGFMLink(node: ParentNode, context: MarkdownSourceContext, recoveredSpans: WeakMap<UnistNode, SourceSpan>): boolean {
|
||||
return isCoveredBySourceRange(node, context.linkRanges, recoveredSpans);
|
||||
}
|
||||
|
||||
function isLiteralEmailLink(node: UnistNode, context: MarkdownSourceContext, recoveredSpans: WeakMap<UnistNode, SourceSpan>): boolean {
|
||||
return (
|
||||
node.type === "link" &&
|
||||
(node as Link).url.startsWith("mailto:") &&
|
||||
!isCoveredBySourceRange(node, context.syntaxLinkRanges, recoveredSpans)
|
||||
);
|
||||
}
|
||||
|
||||
function decodedMarkdownText(source: string): string {
|
||||
return decodeString(source).replaceAll("\0", "<22>");
|
||||
}
|
||||
|
||||
function mergeAdjacentTextNodes(parent: ParentNode, source: string, recoveredSpans: WeakMap<UnistNode, SourceSpan>): void {
|
||||
for (let index = 0; index + 1 < parent.children.length; ) {
|
||||
const left = parent.children[index];
|
||||
const right = parent.children[index + 1];
|
||||
if (left.type !== "text" || right.type !== "text") {
|
||||
index++;
|
||||
continue;
|
||||
}
|
||||
|
||||
const leftSpan = nodeSpan(left, recoveredSpans);
|
||||
const rightSpan = nodeSpan(right, recoveredSpans);
|
||||
if (!leftSpan || !rightSpan || leftSpan.to !== rightSpan.from) {
|
||||
index++;
|
||||
continue;
|
||||
}
|
||||
|
||||
const mergedSpan = { from: leftSpan.from, to: rightSpan.to };
|
||||
const merged = { type: "text", value: decodedMarkdownText(source.slice(mergedSpan.from, mergedSpan.to)) } as Text;
|
||||
recoveredSpans.set(merged, mergedSpan);
|
||||
parent.children.splice(index, 2, merged);
|
||||
}
|
||||
}
|
||||
|
||||
function decodedSourceProjection(source: string, from: number, to: number): { value: string; starts: number[]; ends: number[] } {
|
||||
let value = "";
|
||||
const starts: number[] = [];
|
||||
const ends: number[] = [];
|
||||
|
||||
const append = (raw: string, decoded: string, tokenFrom: number, tokenTo: number) => {
|
||||
const normalized = decoded.replaceAll("\0", "<22>");
|
||||
value += normalized;
|
||||
for (let index = 0; index < normalized.length; index++) {
|
||||
starts.push(raw.length === normalized.length ? tokenFrom + index : tokenFrom);
|
||||
ends.push(raw.length === normalized.length ? tokenFrom + index + 1 : tokenTo);
|
||||
}
|
||||
};
|
||||
|
||||
for (let cursor = from; cursor < to; ) {
|
||||
if (source[cursor] === "\\" && cursor + 1 < to) {
|
||||
const raw = source.slice(cursor, cursor + 2);
|
||||
const decoded = decodeString(raw);
|
||||
if (decoded !== raw) {
|
||||
append(raw, decoded, cursor, cursor + 2);
|
||||
cursor += 2;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
const newNodes = segments.map((segment) => {
|
||||
if (segment.type === "tag") {
|
||||
return createTagNode(segment.value);
|
||||
}
|
||||
return {
|
||||
type: "text",
|
||||
value: segment.value,
|
||||
} as Text;
|
||||
});
|
||||
const reference = decodedMarkdownCharacterReferenceAt(source, cursor, to);
|
||||
if (reference) {
|
||||
append(source.slice(cursor, reference.to), reference.value, cursor, reference.to);
|
||||
cursor = reference.to;
|
||||
continue;
|
||||
}
|
||||
|
||||
const codePoint = source.codePointAt(cursor);
|
||||
const size = codePoint !== undefined && codePoint > 0xffff ? 2 : 1;
|
||||
const raw = source.slice(cursor, Math.min(to, cursor + size));
|
||||
append(raw, raw, cursor, cursor + raw.length);
|
||||
cursor += raw.length;
|
||||
}
|
||||
|
||||
return { value, starts, ends };
|
||||
}
|
||||
|
||||
function recoverTextSpans(
|
||||
parent: ParentNode,
|
||||
source: string,
|
||||
context: MarkdownSourceContext,
|
||||
recoveredSpans: WeakMap<UnistNode, SourceSpan>,
|
||||
): void {
|
||||
const parentSpan = nodeSpan(parent, recoveredSpans);
|
||||
if (!parentSpan) return;
|
||||
|
||||
for (let index = 0; index < parent.children.length; index++) {
|
||||
const child = parent.children[index];
|
||||
if (child.type !== "text" || nodeSpan(child, recoveredSpans)) continue;
|
||||
|
||||
const textNode = child as Text;
|
||||
let lowerBound = parentSpan.from;
|
||||
for (let previous = index - 1; previous >= 0; previous--) {
|
||||
const span = nodeSpan(parent.children[previous], recoveredSpans);
|
||||
if (span) {
|
||||
lowerBound = span.to;
|
||||
break;
|
||||
}
|
||||
}
|
||||
let upperBound = parentSpan.to;
|
||||
for (let next = index + 1; next < parent.children.length; next++) {
|
||||
const span = nodeSpan(parent.children[next], recoveredSpans);
|
||||
if (span) {
|
||||
upperBound = span.from;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Positionless nodes are a remark-gfm recovery artifact. Locate their
|
||||
// decoded value monotonically inside the surrounding structural source;
|
||||
// Markdown delimiters remain in the projection and therefore outside the
|
||||
// recovered span.
|
||||
const projection = decodedSourceProjection(source, lowerBound, upperBound);
|
||||
for (let indexInProjection = projection.value.indexOf(textNode.value); indexInProjection >= 0; ) {
|
||||
const lastIndex = indexInProjection + textNode.value.length - 1;
|
||||
const span = {
|
||||
from: projection.starts[indexInProjection] ?? lowerBound,
|
||||
to: projection.ends[lastIndex] ?? projection.starts[indexInProjection] ?? lowerBound,
|
||||
};
|
||||
if (literalRunsForTextNode(textNode.value, span, source, context.boundaries)) {
|
||||
recoveredSpans.set(child, span);
|
||||
break;
|
||||
}
|
||||
indexInProjection = projection.value.indexOf(textNode.value, indexInProjection + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function synthesizeGFMEmailLinks(
|
||||
parent: ParentNode,
|
||||
source: string,
|
||||
context: MarkdownSourceContext,
|
||||
recoveredSpans: WeakMap<UnistNode, SourceSpan>,
|
||||
): void {
|
||||
for (let index = 0; index < parent.children.length; index++) {
|
||||
const child = parent.children[index];
|
||||
if (child.type !== "text") continue;
|
||||
|
||||
const span = nodeSpan(child, recoveredSpans);
|
||||
if (!span) continue;
|
||||
const emailRanges = context.emailRanges.filter((range) => span.from <= range.from && span.to >= range.to);
|
||||
if (emailRanges.length === 0) continue;
|
||||
|
||||
const replacement: UnistNode[] = [];
|
||||
let cursor = span.from;
|
||||
for (const range of emailRanges) {
|
||||
if (cursor < range.from) {
|
||||
const text = { type: "text", value: decodedMarkdownText(source.slice(cursor, range.from)) } as Text;
|
||||
recoveredSpans.set(text, { from: cursor, to: range.from });
|
||||
replacement.push(text);
|
||||
}
|
||||
|
||||
const value = range.value;
|
||||
const text = { type: "text", value } as Text;
|
||||
const link = { type: "link", title: null, url: `mailto:${value}`, children: [text] } as Link;
|
||||
recoveredSpans.set(text, range);
|
||||
recoveredSpans.set(link, range);
|
||||
replacement.push(link as UnistNode);
|
||||
cursor = range.to;
|
||||
}
|
||||
if (cursor < span.to) {
|
||||
const text = { type: "text", value: decodedMarkdownText(source.slice(cursor, span.to)) } as Text;
|
||||
recoveredSpans.set(text, { from: cursor, to: span.to });
|
||||
replacement.push(text);
|
||||
}
|
||||
|
||||
parent.children.splice(index, 1, ...replacement);
|
||||
index += replacement.length - 1;
|
||||
}
|
||||
}
|
||||
|
||||
function reconcileUnderscoreEmailEmphasis(
|
||||
parent: ParentNode,
|
||||
source: string,
|
||||
context: MarkdownSourceContext,
|
||||
recoveredSpans: WeakMap<UnistNode, SourceSpan>,
|
||||
): void {
|
||||
for (const range of context.emphasisRanges) {
|
||||
const delimiter = "_".repeat(range.delimiterWidth);
|
||||
if (
|
||||
source.slice(range.from, range.from + range.delimiterWidth) !== delimiter ||
|
||||
source.slice(range.to - range.delimiterWidth, range.to) !== delimiter
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
const innerFrom = range.from + range.delimiterWidth;
|
||||
const innerTo = range.to - range.delimiterWidth;
|
||||
if (
|
||||
!context.emailRanges.some(
|
||||
(emailRange) =>
|
||||
emailRange.from >= innerFrom &&
|
||||
emailRange.to <= innerTo &&
|
||||
[...source.slice(innerFrom, emailRange.from)].every((character) => character === "_"),
|
||||
)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const firstIndex = parent.children.findIndex((child) => {
|
||||
const span = nodeSpan(child, recoveredSpans);
|
||||
return !!span && span.from <= range.from && span.to >= innerFrom;
|
||||
});
|
||||
if (firstIndex < 0) continue;
|
||||
const lastIndex = parent.children.findIndex((child, index) => {
|
||||
const span = nodeSpan(child, recoveredSpans);
|
||||
return index >= firstIndex && !!span && span.from <= innerTo && span.to >= range.to;
|
||||
});
|
||||
if (lastIndex <= firstIndex) continue;
|
||||
|
||||
const first = parent.children[firstIndex];
|
||||
const last = parent.children[lastIndex];
|
||||
const firstSpan = nodeSpan(first, recoveredSpans);
|
||||
const lastSpan = nodeSpan(last, recoveredSpans);
|
||||
if (first.type !== "text" || last.type !== "text" || !firstSpan || !lastSpan) continue;
|
||||
|
||||
const textNode = (from: number, to: number): Text => {
|
||||
const text = { type: "text", value: decodedMarkdownText(source.slice(from, to)) } as Text;
|
||||
recoveredSpans.set(text, { from, to });
|
||||
return text;
|
||||
};
|
||||
const emphasisChildren: UnistNode[] = [];
|
||||
if (innerFrom < firstSpan.to) emphasisChildren.push(textNode(innerFrom, Math.min(firstSpan.to, innerTo)));
|
||||
emphasisChildren.push(...parent.children.slice(firstIndex + 1, lastIndex));
|
||||
if (lastSpan.from < innerTo) emphasisChildren.push(textNode(Math.max(lastSpan.from, innerFrom), innerTo));
|
||||
|
||||
const emphasis = { type: range.delimiterWidth === 2 ? "strong" : "emphasis", children: emphasisChildren } as ParentNode;
|
||||
recoveredSpans.set(emphasis, range);
|
||||
const replacement: UnistNode[] = [];
|
||||
if (firstSpan.from < range.from) replacement.push(textNode(firstSpan.from, range.from));
|
||||
replacement.push(emphasis);
|
||||
if (range.to < lastSpan.to) replacement.push(textNode(range.to, lastSpan.to));
|
||||
parent.children.splice(firstIndex, lastIndex - firstIndex + 1, ...replacement);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* remark-gfm accepts links beyond written GFM and can split or de-position the
|
||||
* surrounding text. Reconcile that drift before scanning so original source
|
||||
* ranges, rather than incidental mdast child boundaries, define literal runs.
|
||||
*/
|
||||
function reconcileTextNodes(
|
||||
parent: ParentNode,
|
||||
source: string,
|
||||
context: MarkdownSourceContext,
|
||||
recoveredSpans: WeakMap<UnistNode, SourceSpan>,
|
||||
): void {
|
||||
for (let index = 0; index < parent.children.length; index++) {
|
||||
const child = parent.children[index];
|
||||
|
||||
if (isParentNode(child) && TRANSPARENT_PARENT_TYPES.has(child.type)) {
|
||||
reconcileTextNodes(child, source, context, recoveredSpans);
|
||||
} else if (
|
||||
child.type === "link" &&
|
||||
isParentNode(child) &&
|
||||
(isLiteralEmailLink(child, context, recoveredSpans) || !isWrittenGFMLink(child, context, recoveredSpans))
|
||||
) {
|
||||
// Rebuild literal emails from canonical source ranges, and unwrap only
|
||||
// remark-gfm's broader URL forms. Written GFM URL links stay opaque.
|
||||
const span = nodeSpan(child, recoveredSpans);
|
||||
if (span) {
|
||||
const textNode = { type: "text", value: decodedMarkdownText(source.slice(span.from, span.to)) } as Text;
|
||||
recoveredSpans.set(textNode, span);
|
||||
parent.children.splice(index, 1, textNode);
|
||||
} else {
|
||||
parent.children.splice(index, 1, ...child.children);
|
||||
}
|
||||
index--;
|
||||
}
|
||||
}
|
||||
|
||||
mergeAdjacentTextNodes(parent, source, recoveredSpans);
|
||||
recoverTextSpans(parent, source, context, recoveredSpans);
|
||||
mergeAdjacentTextNodes(parent, source, recoveredSpans);
|
||||
synthesizeGFMEmailLinks(parent, source, context, recoveredSpans);
|
||||
reconcileUnderscoreEmailEmphasis(parent, source, context, recoveredSpans);
|
||||
}
|
||||
|
||||
function transformTagTextNodes(
|
||||
parent: ParentNode,
|
||||
source: string,
|
||||
context: MarkdownSourceContext,
|
||||
recoveredSpans: WeakMap<UnistNode, SourceSpan>,
|
||||
): void {
|
||||
for (let index = 0; index < parent.children.length; index++) {
|
||||
const child = parent.children[index];
|
||||
|
||||
if (child.type === "text") {
|
||||
const textNode = child as Text;
|
||||
const segments = segmentsForTextNode(textNode.value, nodeSpan(textNode, recoveredSpans), source, context.boundaries);
|
||||
if (!segments || segments.every((segment) => segment.type === "text")) continue;
|
||||
|
||||
const newNodes = segments.map((segment) =>
|
||||
segment.type === "tag" ? createTagNode(segment.value, segment.source) : ({ type: "text", value: segment.value } as Text),
|
||||
);
|
||||
parent.children.splice(index, 1, ...(newNodes as UnistNode[]));
|
||||
index += newNodes.length - 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (isParentNode(child)) {
|
||||
transformTagTextNodes(child, childInsideLink, source);
|
||||
if (isParentNode(child) && TRANSPARENT_PARENT_TYPES.has(child.type)) {
|
||||
transformTagTextNodes(child, source, context, recoveredSpans);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -186,7 +712,11 @@ type VFileLike = { value?: string | Uint8Array };
|
|||
|
||||
export const remarkTag = () => {
|
||||
return (tree: Root, file: VFileLike) => {
|
||||
const source = typeof file?.value === "string" ? file.value : "";
|
||||
transformTagTextNodes(tree as ParentNode, false, source);
|
||||
const rawSource = typeof file.value === "string" ? file.value : "";
|
||||
const source = rawSource.startsWith("\uFEFF") ? rawSource.slice(1) : rawSource;
|
||||
const context = markdownSourceContext(source);
|
||||
const recoveredSpans = new WeakMap<UnistNode, SourceSpan>();
|
||||
reconcileTextNodes(tree as ParentNode, source, context, recoveredSpans);
|
||||
transformTagTextNodes(tree as ParentNode, source, context, recoveredSpans);
|
||||
};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,37 +1,201 @@
|
|||
/**
|
||||
* The single source of truth for memos' `#tag` lexing grammar, shared by the
|
||||
* editor and the read-only renderer (utils/remark-plugins/remark-tag.ts) so they
|
||||
* can't drift.
|
||||
*
|
||||
* A tag character is any Unicode letter, mark, number, or symbol, plus
|
||||
* `_ - / &`. The mark class (`\p{M}`) keeps combining marks — Indic vowel
|
||||
* signs, Arabic harakat, Hebrew niqqud, decomposed accents — attached to the
|
||||
* base letters they belong to, so a tag like `#കവിത` isn't cut short at its
|
||||
* first vowel sign.
|
||||
* A tag run is capped at MAX_TAG_LENGTH characters.
|
||||
*/
|
||||
export const TAG_CHAR_CLASS = "[\\p{L}\\p{M}\\p{N}\\p{S}_\\-/&]";
|
||||
import { COMBINING_MARK_RANGES, DEFAULT_IGNORABLE_RANGES, FULLY_QUALIFIED_EMOJI, XID_CONTINUE_RANGES } from "@/utils/tag-unicode-data";
|
||||
|
||||
export const MAX_TAG_LENGTH = 100;
|
||||
|
||||
/**
|
||||
* Regex source for a *capped* tag run: 1..MAX_TAG_LENGTH tag characters,
|
||||
* refusing to match when a (MAX+1)-th tag character would follow (an over-long
|
||||
* run is not a tag). Embed with the `u` flag.
|
||||
*
|
||||
* The two halves of the round-trip build on this same source so they can't
|
||||
* disagree on what counts as a tag: the editor's input rule + tokenizer match
|
||||
* `#(${TAG_RUN})`, and the serialize-escape (tagMarkdown.ts) escapes a `#`
|
||||
* followed by `(?=${TAG_RUN})`.
|
||||
*/
|
||||
export const TAG_RUN = `${TAG_CHAR_CLASS}{1,${MAX_TAG_LENGTH}}(?!${TAG_CHAR_CLASS})`;
|
||||
|
||||
// Matches exactly one tag character. The `u` flag makes the class match whole
|
||||
// code points, so astral-plane symbols (emoji et al.) are tested intact rather
|
||||
// than as lone surrogates.
|
||||
const SINGLE_TAG_CHAR = new RegExp(`^${TAG_CHAR_CLASS}$`, "u");
|
||||
|
||||
/** Whether a single character (one code point) is allowed inside a tag. */
|
||||
export function isTagChar(char: string): boolean {
|
||||
return SINGLE_TAG_CHAR.test(char);
|
||||
/** A recognized `#tag` source span and the identifier emitted from it. */
|
||||
export interface TagMatch {
|
||||
/** UTF-16 offsets into the scanned source, including the `#` introducer. */
|
||||
from: number;
|
||||
to: number;
|
||||
/** Exact source spelling after the introducer, including ignored code points. */
|
||||
source: string;
|
||||
/** Emitted tag identifier, excluding the introducer and ignored code points. */
|
||||
value: string;
|
||||
}
|
||||
|
||||
interface EmojiTrieNode {
|
||||
children: Map<number, EmojiTrieNode>;
|
||||
terminal: boolean;
|
||||
}
|
||||
|
||||
const emojiTrie: EmojiTrieNode = { children: new Map(), terminal: false };
|
||||
for (const emoji of FULLY_QUALIFIED_EMOJI) {
|
||||
let node = emojiTrie;
|
||||
for (const codePoint of emoji) {
|
||||
const value = codePoint.codePointAt(0) ?? 0;
|
||||
let child = node.children.get(value);
|
||||
if (!child) {
|
||||
child = { children: new Map(), terminal: false };
|
||||
node.children.set(value, child);
|
||||
}
|
||||
node = child;
|
||||
}
|
||||
node.terminal = true;
|
||||
}
|
||||
|
||||
function codePointAt(source: string, index: number): string {
|
||||
const value = source.codePointAt(index);
|
||||
return value === undefined ? "" : String.fromCodePoint(value);
|
||||
}
|
||||
|
||||
function isInRanges(codePoint: number, ranges: readonly number[]): boolean {
|
||||
let low = 0;
|
||||
let high = ranges.length / 2 - 1;
|
||||
while (low <= high) {
|
||||
const middle = Math.floor((low + high) / 2);
|
||||
const from = ranges[middle * 2];
|
||||
const to = ranges[middle * 2 + 1];
|
||||
if (codePoint < from) high = middle - 1;
|
||||
else if (codePoint > to) low = middle + 1;
|
||||
else return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Match the longest fully-qualified Emoji 17 sequence at a UTF-16 offset. */
|
||||
function emojiAt(source: string, index: number, limit: number): string | undefined {
|
||||
let node = emojiTrie;
|
||||
let cursor = index;
|
||||
let longestEnd = -1;
|
||||
while (cursor < limit) {
|
||||
const codePoint = source.codePointAt(cursor);
|
||||
if (codePoint === undefined) break;
|
||||
const child = node.children.get(codePoint);
|
||||
if (!child) break;
|
||||
cursor += codePoint > 0xffff ? 2 : 1;
|
||||
if (cursor > limit) break;
|
||||
node = child;
|
||||
if (node.terminal) longestEnd = cursor;
|
||||
}
|
||||
return longestEnd < 0 ? undefined : source.slice(index, longestEnd);
|
||||
}
|
||||
|
||||
/** Whether the `#` at a UTF-16 offset is an introducer rather than emoji. */
|
||||
export function isTagIntroducerAt(source: string, index: number, limit = source.length): boolean {
|
||||
return index >= 0 && index < limit && source[index] === "#" && !emojiAt(source, index, limit);
|
||||
}
|
||||
|
||||
function isDefaultIgnorable(value: string): boolean {
|
||||
return isInRanges(value.codePointAt(0) ?? 0, DEFAULT_IGNORABLE_RANGES);
|
||||
}
|
||||
|
||||
function isXIDContinue(value: string): boolean {
|
||||
return isInRanges(value.codePointAt(0) ?? 0, XID_CONTINUE_RANGES) && !isDefaultIgnorable(value);
|
||||
}
|
||||
|
||||
function isCombiningMark(value: string): boolean {
|
||||
return isInRanges(value.codePointAt(0) ?? 0, COMBINING_MARK_RANGES);
|
||||
}
|
||||
|
||||
function isExtensionUnit(value: string): boolean {
|
||||
return value === "-" || value === "+" || value === "&";
|
||||
}
|
||||
|
||||
interface SegmentMatch {
|
||||
to: number;
|
||||
value: string;
|
||||
}
|
||||
|
||||
function scanSegment(source: string, from: number, limit: number): SegmentMatch | undefined {
|
||||
let index = from;
|
||||
|
||||
// These code points belong to the source spelling but do not make a segment
|
||||
// non-empty and do not contribute to its emitted value.
|
||||
while (index < limit) {
|
||||
const codePoint = codePointAt(source, index);
|
||||
if (isDefaultIgnorable(codePoint) || (isXIDContinue(codePoint) && isCombiningMark(codePoint))) {
|
||||
index += codePoint.length;
|
||||
continue;
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
let value = "";
|
||||
const starterEmoji = emojiAt(source, index, limit);
|
||||
if (starterEmoji) {
|
||||
value = starterEmoji;
|
||||
index += starterEmoji.length;
|
||||
} else {
|
||||
const starter = codePointAt(source, index);
|
||||
if (!starter || (!(isXIDContinue(starter) && !isCombiningMark(starter)) && !isExtensionUnit(starter))) return undefined;
|
||||
value = starter;
|
||||
index += starter.length;
|
||||
}
|
||||
|
||||
while (index < limit) {
|
||||
const emoji = emojiAt(source, index, limit);
|
||||
if (emoji) {
|
||||
value += emoji;
|
||||
index += emoji.length;
|
||||
continue;
|
||||
}
|
||||
|
||||
const codePoint = codePointAt(source, index);
|
||||
if (isDefaultIgnorable(codePoint)) {
|
||||
index += codePoint.length;
|
||||
continue;
|
||||
}
|
||||
if (isXIDContinue(codePoint) || isExtensionUnit(codePoint)) {
|
||||
value += codePoint;
|
||||
index += codePoint.length;
|
||||
continue;
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
return { to: index, value };
|
||||
}
|
||||
|
||||
/**
|
||||
* Scan one tag candidate beginning at `from`.
|
||||
*
|
||||
* Offsets are UTF-16 offsets so callers can use them directly with Markdown
|
||||
* source positions and CodeMirror ranges.
|
||||
*/
|
||||
export function scanTagAt(source: string, from: number, limit = source.length): TagMatch | undefined {
|
||||
if (!isTagIntroducerAt(source, from, limit)) return undefined;
|
||||
|
||||
const firstSegment = scanSegment(source, from + 1, limit);
|
||||
if (!firstSegment) return undefined;
|
||||
|
||||
let to = firstSegment.to;
|
||||
let value = firstSegment.value;
|
||||
while (to < limit && source[to] === "/") {
|
||||
const nextSegment = scanSegment(source, to + 1, limit);
|
||||
if (!nextSegment) break;
|
||||
value += `/${nextSegment.value}`;
|
||||
to = nextSegment.to;
|
||||
}
|
||||
|
||||
return {
|
||||
from,
|
||||
to,
|
||||
source: source.slice(from + 1, to),
|
||||
value,
|
||||
};
|
||||
}
|
||||
|
||||
/** Enumerate every tag candidate in one eligible literal-source run. */
|
||||
export function findTagMatches(source: string, from = 0, limit = source.length): TagMatch[] {
|
||||
const matches: TagMatch[] = [];
|
||||
let index = from;
|
||||
|
||||
while (index < limit) {
|
||||
const emoji = emojiAt(source, index, limit);
|
||||
if (emoji) {
|
||||
index += emoji.length;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (source[index] === "#") {
|
||||
const match = scanTagAt(source, index, limit);
|
||||
if (match) {
|
||||
matches.push(match);
|
||||
index = match.to;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
index += codePointAt(source, index).length;
|
||||
}
|
||||
|
||||
return matches;
|
||||
}
|
||||
|
|
|
|||
538
web/src/utils/tag-unicode-data.ts
Normal file
538
web/src/utils/tag-unicode-data.ts
Normal file
|
|
@ -0,0 +1,538 @@
|
|||
// Generated by scripts/generate-tag-unicode-data.mjs. Do not edit.
|
||||
// Unicode and Emoji 17.0.0 data used by Memos tag recognition.
|
||||
// Unicode data SHA-256: a5ac7d15cb353b0a0fd6571da68ec89a5681d3b1d68a74ac1c6e583b3761ea66
|
||||
// biome-ignore-all format: generated data is kept in compact source order
|
||||
|
||||
export const XID_CONTINUE_RANGES: readonly number[] = [
|
||||
0x30, 0x39, 0x41, 0x5a, 0x5f, 0x5f, 0x61, 0x7a, 0xaa, 0xaa, 0xb5, 0xb5,
|
||||
0xb7, 0xb7, 0xba, 0xba, 0xc0, 0xd6, 0xd8, 0xf6, 0xf8, 0x2c1, 0x2c6, 0x2d1,
|
||||
0x2e0, 0x2e4, 0x2ec, 0x2ec, 0x2ee, 0x2ee, 0x300, 0x374, 0x376, 0x377, 0x37b, 0x37d,
|
||||
0x37f, 0x37f, 0x386, 0x38a, 0x38c, 0x38c, 0x38e, 0x3a1, 0x3a3, 0x3f5, 0x3f7, 0x481,
|
||||
0x483, 0x487, 0x48a, 0x52f, 0x531, 0x556, 0x559, 0x559, 0x560, 0x588, 0x591, 0x5bd,
|
||||
0x5bf, 0x5bf, 0x5c1, 0x5c2, 0x5c4, 0x5c5, 0x5c7, 0x5c7, 0x5d0, 0x5ea, 0x5ef, 0x5f2,
|
||||
0x610, 0x61a, 0x620, 0x669, 0x66e, 0x6d3, 0x6d5, 0x6dc, 0x6df, 0x6e8, 0x6ea, 0x6fc,
|
||||
0x6ff, 0x6ff, 0x710, 0x74a, 0x74d, 0x7b1, 0x7c0, 0x7f5, 0x7fa, 0x7fa, 0x7fd, 0x7fd,
|
||||
0x800, 0x82d, 0x840, 0x85b, 0x860, 0x86a, 0x870, 0x887, 0x889, 0x88f, 0x897, 0x8e1,
|
||||
0x8e3, 0x963, 0x966, 0x96f, 0x971, 0x983, 0x985, 0x98c, 0x98f, 0x990, 0x993, 0x9a8,
|
||||
0x9aa, 0x9b0, 0x9b2, 0x9b2, 0x9b6, 0x9b9, 0x9bc, 0x9c4, 0x9c7, 0x9c8, 0x9cb, 0x9ce,
|
||||
0x9d7, 0x9d7, 0x9dc, 0x9dd, 0x9df, 0x9e3, 0x9e6, 0x9f1, 0x9fc, 0x9fc, 0x9fe, 0x9fe,
|
||||
0xa01, 0xa03, 0xa05, 0xa0a, 0xa0f, 0xa10, 0xa13, 0xa28, 0xa2a, 0xa30, 0xa32, 0xa33,
|
||||
0xa35, 0xa36, 0xa38, 0xa39, 0xa3c, 0xa3c, 0xa3e, 0xa42, 0xa47, 0xa48, 0xa4b, 0xa4d,
|
||||
0xa51, 0xa51, 0xa59, 0xa5c, 0xa5e, 0xa5e, 0xa66, 0xa75, 0xa81, 0xa83, 0xa85, 0xa8d,
|
||||
0xa8f, 0xa91, 0xa93, 0xaa8, 0xaaa, 0xab0, 0xab2, 0xab3, 0xab5, 0xab9, 0xabc, 0xac5,
|
||||
0xac7, 0xac9, 0xacb, 0xacd, 0xad0, 0xad0, 0xae0, 0xae3, 0xae6, 0xaef, 0xaf9, 0xaff,
|
||||
0xb01, 0xb03, 0xb05, 0xb0c, 0xb0f, 0xb10, 0xb13, 0xb28, 0xb2a, 0xb30, 0xb32, 0xb33,
|
||||
0xb35, 0xb39, 0xb3c, 0xb44, 0xb47, 0xb48, 0xb4b, 0xb4d, 0xb55, 0xb57, 0xb5c, 0xb5d,
|
||||
0xb5f, 0xb63, 0xb66, 0xb6f, 0xb71, 0xb71, 0xb82, 0xb83, 0xb85, 0xb8a, 0xb8e, 0xb90,
|
||||
0xb92, 0xb95, 0xb99, 0xb9a, 0xb9c, 0xb9c, 0xb9e, 0xb9f, 0xba3, 0xba4, 0xba8, 0xbaa,
|
||||
0xbae, 0xbb9, 0xbbe, 0xbc2, 0xbc6, 0xbc8, 0xbca, 0xbcd, 0xbd0, 0xbd0, 0xbd7, 0xbd7,
|
||||
0xbe6, 0xbef, 0xc00, 0xc0c, 0xc0e, 0xc10, 0xc12, 0xc28, 0xc2a, 0xc39, 0xc3c, 0xc44,
|
||||
0xc46, 0xc48, 0xc4a, 0xc4d, 0xc55, 0xc56, 0xc58, 0xc5a, 0xc5c, 0xc5d, 0xc60, 0xc63,
|
||||
0xc66, 0xc6f, 0xc80, 0xc83, 0xc85, 0xc8c, 0xc8e, 0xc90, 0xc92, 0xca8, 0xcaa, 0xcb3,
|
||||
0xcb5, 0xcb9, 0xcbc, 0xcc4, 0xcc6, 0xcc8, 0xcca, 0xccd, 0xcd5, 0xcd6, 0xcdc, 0xcde,
|
||||
0xce0, 0xce3, 0xce6, 0xcef, 0xcf1, 0xcf3, 0xd00, 0xd0c, 0xd0e, 0xd10, 0xd12, 0xd44,
|
||||
0xd46, 0xd48, 0xd4a, 0xd4e, 0xd54, 0xd57, 0xd5f, 0xd63, 0xd66, 0xd6f, 0xd7a, 0xd7f,
|
||||
0xd81, 0xd83, 0xd85, 0xd96, 0xd9a, 0xdb1, 0xdb3, 0xdbb, 0xdbd, 0xdbd, 0xdc0, 0xdc6,
|
||||
0xdca, 0xdca, 0xdcf, 0xdd4, 0xdd6, 0xdd6, 0xdd8, 0xddf, 0xde6, 0xdef, 0xdf2, 0xdf3,
|
||||
0xe01, 0xe3a, 0xe40, 0xe4e, 0xe50, 0xe59, 0xe81, 0xe82, 0xe84, 0xe84, 0xe86, 0xe8a,
|
||||
0xe8c, 0xea3, 0xea5, 0xea5, 0xea7, 0xebd, 0xec0, 0xec4, 0xec6, 0xec6, 0xec8, 0xece,
|
||||
0xed0, 0xed9, 0xedc, 0xedf, 0xf00, 0xf00, 0xf18, 0xf19, 0xf20, 0xf29, 0xf35, 0xf35,
|
||||
0xf37, 0xf37, 0xf39, 0xf39, 0xf3e, 0xf47, 0xf49, 0xf6c, 0xf71, 0xf84, 0xf86, 0xf97,
|
||||
0xf99, 0xfbc, 0xfc6, 0xfc6, 0x1000, 0x1049, 0x1050, 0x109d, 0x10a0, 0x10c5, 0x10c7, 0x10c7,
|
||||
0x10cd, 0x10cd, 0x10d0, 0x10fa, 0x10fc, 0x1248, 0x124a, 0x124d, 0x1250, 0x1256, 0x1258, 0x1258,
|
||||
0x125a, 0x125d, 0x1260, 0x1288, 0x128a, 0x128d, 0x1290, 0x12b0, 0x12b2, 0x12b5, 0x12b8, 0x12be,
|
||||
0x12c0, 0x12c0, 0x12c2, 0x12c5, 0x12c8, 0x12d6, 0x12d8, 0x1310, 0x1312, 0x1315, 0x1318, 0x135a,
|
||||
0x135d, 0x135f, 0x1369, 0x1371, 0x1380, 0x138f, 0x13a0, 0x13f5, 0x13f8, 0x13fd, 0x1401, 0x166c,
|
||||
0x166f, 0x167f, 0x1681, 0x169a, 0x16a0, 0x16ea, 0x16ee, 0x16f8, 0x1700, 0x1715, 0x171f, 0x1734,
|
||||
0x1740, 0x1753, 0x1760, 0x176c, 0x176e, 0x1770, 0x1772, 0x1773, 0x1780, 0x17d3, 0x17d7, 0x17d7,
|
||||
0x17dc, 0x17dd, 0x17e0, 0x17e9, 0x180b, 0x180d, 0x180f, 0x1819, 0x1820, 0x1878, 0x1880, 0x18aa,
|
||||
0x18b0, 0x18f5, 0x1900, 0x191e, 0x1920, 0x192b, 0x1930, 0x193b, 0x1946, 0x196d, 0x1970, 0x1974,
|
||||
0x1980, 0x19ab, 0x19b0, 0x19c9, 0x19d0, 0x19da, 0x1a00, 0x1a1b, 0x1a20, 0x1a5e, 0x1a60, 0x1a7c,
|
||||
0x1a7f, 0x1a89, 0x1a90, 0x1a99, 0x1aa7, 0x1aa7, 0x1ab0, 0x1abd, 0x1abf, 0x1add, 0x1ae0, 0x1aeb,
|
||||
0x1b00, 0x1b4c, 0x1b50, 0x1b59, 0x1b6b, 0x1b73, 0x1b80, 0x1bf3, 0x1c00, 0x1c37, 0x1c40, 0x1c49,
|
||||
0x1c4d, 0x1c7d, 0x1c80, 0x1c8a, 0x1c90, 0x1cba, 0x1cbd, 0x1cbf, 0x1cd0, 0x1cd2, 0x1cd4, 0x1cfa,
|
||||
0x1d00, 0x1f15, 0x1f18, 0x1f1d, 0x1f20, 0x1f45, 0x1f48, 0x1f4d, 0x1f50, 0x1f57, 0x1f59, 0x1f59,
|
||||
0x1f5b, 0x1f5b, 0x1f5d, 0x1f5d, 0x1f5f, 0x1f7d, 0x1f80, 0x1fb4, 0x1fb6, 0x1fbc, 0x1fbe, 0x1fbe,
|
||||
0x1fc2, 0x1fc4, 0x1fc6, 0x1fcc, 0x1fd0, 0x1fd3, 0x1fd6, 0x1fdb, 0x1fe0, 0x1fec, 0x1ff2, 0x1ff4,
|
||||
0x1ff6, 0x1ffc, 0x200c, 0x200d, 0x203f, 0x2040, 0x2054, 0x2054, 0x2071, 0x2071, 0x207f, 0x207f,
|
||||
0x2090, 0x209c, 0x20d0, 0x20dc, 0x20e1, 0x20e1, 0x20e5, 0x20f0, 0x2102, 0x2102, 0x2107, 0x2107,
|
||||
0x210a, 0x2113, 0x2115, 0x2115, 0x2118, 0x211d, 0x2124, 0x2124, 0x2126, 0x2126, 0x2128, 0x2128,
|
||||
0x212a, 0x2139, 0x213c, 0x213f, 0x2145, 0x2149, 0x214e, 0x214e, 0x2160, 0x2188, 0x2c00, 0x2ce4,
|
||||
0x2ceb, 0x2cf3, 0x2d00, 0x2d25, 0x2d27, 0x2d27, 0x2d2d, 0x2d2d, 0x2d30, 0x2d67, 0x2d6f, 0x2d6f,
|
||||
0x2d7f, 0x2d96, 0x2da0, 0x2da6, 0x2da8, 0x2dae, 0x2db0, 0x2db6, 0x2db8, 0x2dbe, 0x2dc0, 0x2dc6,
|
||||
0x2dc8, 0x2dce, 0x2dd0, 0x2dd6, 0x2dd8, 0x2dde, 0x2de0, 0x2dff, 0x3005, 0x3007, 0x3021, 0x302f,
|
||||
0x3031, 0x3035, 0x3038, 0x303c, 0x3041, 0x3096, 0x3099, 0x309a, 0x309d, 0x309f, 0x30a1, 0x30ff,
|
||||
0x3105, 0x312f, 0x3131, 0x318e, 0x31a0, 0x31bf, 0x31f0, 0x31ff, 0x3400, 0x4dbf, 0x4e00, 0xa48c,
|
||||
0xa4d0, 0xa4fd, 0xa500, 0xa60c, 0xa610, 0xa62b, 0xa640, 0xa66f, 0xa674, 0xa67d, 0xa67f, 0xa6f1,
|
||||
0xa717, 0xa71f, 0xa722, 0xa788, 0xa78b, 0xa7dc, 0xa7f1, 0xa827, 0xa82c, 0xa82c, 0xa840, 0xa873,
|
||||
0xa880, 0xa8c5, 0xa8d0, 0xa8d9, 0xa8e0, 0xa8f7, 0xa8fb, 0xa8fb, 0xa8fd, 0xa92d, 0xa930, 0xa953,
|
||||
0xa960, 0xa97c, 0xa980, 0xa9c0, 0xa9cf, 0xa9d9, 0xa9e0, 0xa9fe, 0xaa00, 0xaa36, 0xaa40, 0xaa4d,
|
||||
0xaa50, 0xaa59, 0xaa60, 0xaa76, 0xaa7a, 0xaac2, 0xaadb, 0xaadd, 0xaae0, 0xaaef, 0xaaf2, 0xaaf6,
|
||||
0xab01, 0xab06, 0xab09, 0xab0e, 0xab11, 0xab16, 0xab20, 0xab26, 0xab28, 0xab2e, 0xab30, 0xab5a,
|
||||
0xab5c, 0xab69, 0xab70, 0xabea, 0xabec, 0xabed, 0xabf0, 0xabf9, 0xac00, 0xd7a3, 0xd7b0, 0xd7c6,
|
||||
0xd7cb, 0xd7fb, 0xf900, 0xfa6d, 0xfa70, 0xfad9, 0xfb00, 0xfb06, 0xfb13, 0xfb17, 0xfb1d, 0xfb28,
|
||||
0xfb2a, 0xfb36, 0xfb38, 0xfb3c, 0xfb3e, 0xfb3e, 0xfb40, 0xfb41, 0xfb43, 0xfb44, 0xfb46, 0xfbb1,
|
||||
0xfbd3, 0xfc5d, 0xfc64, 0xfd3d, 0xfd50, 0xfd8f, 0xfd92, 0xfdc7, 0xfdf0, 0xfdf9, 0xfe00, 0xfe0f,
|
||||
0xfe20, 0xfe2f, 0xfe33, 0xfe34, 0xfe4d, 0xfe4f, 0xfe71, 0xfe71, 0xfe73, 0xfe73, 0xfe77, 0xfe77,
|
||||
0xfe79, 0xfe79, 0xfe7b, 0xfe7b, 0xfe7d, 0xfe7d, 0xfe7f, 0xfefc, 0xff10, 0xff19, 0xff21, 0xff3a,
|
||||
0xff3f, 0xff3f, 0xff41, 0xff5a, 0xff65, 0xffbe, 0xffc2, 0xffc7, 0xffca, 0xffcf, 0xffd2, 0xffd7,
|
||||
0xffda, 0xffdc, 0x10000, 0x1000b, 0x1000d, 0x10026, 0x10028, 0x1003a, 0x1003c, 0x1003d, 0x1003f, 0x1004d,
|
||||
0x10050, 0x1005d, 0x10080, 0x100fa, 0x10140, 0x10174, 0x101fd, 0x101fd, 0x10280, 0x1029c, 0x102a0, 0x102d0,
|
||||
0x102e0, 0x102e0, 0x10300, 0x1031f, 0x1032d, 0x1034a, 0x10350, 0x1037a, 0x10380, 0x1039d, 0x103a0, 0x103c3,
|
||||
0x103c8, 0x103cf, 0x103d1, 0x103d5, 0x10400, 0x1049d, 0x104a0, 0x104a9, 0x104b0, 0x104d3, 0x104d8, 0x104fb,
|
||||
0x10500, 0x10527, 0x10530, 0x10563, 0x10570, 0x1057a, 0x1057c, 0x1058a, 0x1058c, 0x10592, 0x10594, 0x10595,
|
||||
0x10597, 0x105a1, 0x105a3, 0x105b1, 0x105b3, 0x105b9, 0x105bb, 0x105bc, 0x105c0, 0x105f3, 0x10600, 0x10736,
|
||||
0x10740, 0x10755, 0x10760, 0x10767, 0x10780, 0x10785, 0x10787, 0x107b0, 0x107b2, 0x107ba, 0x10800, 0x10805,
|
||||
0x10808, 0x10808, 0x1080a, 0x10835, 0x10837, 0x10838, 0x1083c, 0x1083c, 0x1083f, 0x10855, 0x10860, 0x10876,
|
||||
0x10880, 0x1089e, 0x108e0, 0x108f2, 0x108f4, 0x108f5, 0x10900, 0x10915, 0x10920, 0x10939, 0x10940, 0x10959,
|
||||
0x10980, 0x109b7, 0x109be, 0x109bf, 0x10a00, 0x10a03, 0x10a05, 0x10a06, 0x10a0c, 0x10a13, 0x10a15, 0x10a17,
|
||||
0x10a19, 0x10a35, 0x10a38, 0x10a3a, 0x10a3f, 0x10a3f, 0x10a60, 0x10a7c, 0x10a80, 0x10a9c, 0x10ac0, 0x10ac7,
|
||||
0x10ac9, 0x10ae6, 0x10b00, 0x10b35, 0x10b40, 0x10b55, 0x10b60, 0x10b72, 0x10b80, 0x10b91, 0x10c00, 0x10c48,
|
||||
0x10c80, 0x10cb2, 0x10cc0, 0x10cf2, 0x10d00, 0x10d27, 0x10d30, 0x10d39, 0x10d40, 0x10d65, 0x10d69, 0x10d6d,
|
||||
0x10d6f, 0x10d85, 0x10e80, 0x10ea9, 0x10eab, 0x10eac, 0x10eb0, 0x10eb1, 0x10ec2, 0x10ec7, 0x10efa, 0x10f1c,
|
||||
0x10f27, 0x10f27, 0x10f30, 0x10f50, 0x10f70, 0x10f85, 0x10fb0, 0x10fc4, 0x10fe0, 0x10ff6, 0x11000, 0x11046,
|
||||
0x11066, 0x11075, 0x1107f, 0x110ba, 0x110c2, 0x110c2, 0x110d0, 0x110e8, 0x110f0, 0x110f9, 0x11100, 0x11134,
|
||||
0x11136, 0x1113f, 0x11144, 0x11147, 0x11150, 0x11173, 0x11176, 0x11176, 0x11180, 0x111c4, 0x111c9, 0x111cc,
|
||||
0x111ce, 0x111da, 0x111dc, 0x111dc, 0x11200, 0x11211, 0x11213, 0x11237, 0x1123e, 0x11241, 0x11280, 0x11286,
|
||||
0x11288, 0x11288, 0x1128a, 0x1128d, 0x1128f, 0x1129d, 0x1129f, 0x112a8, 0x112b0, 0x112ea, 0x112f0, 0x112f9,
|
||||
0x11300, 0x11303, 0x11305, 0x1130c, 0x1130f, 0x11310, 0x11313, 0x11328, 0x1132a, 0x11330, 0x11332, 0x11333,
|
||||
0x11335, 0x11339, 0x1133b, 0x11344, 0x11347, 0x11348, 0x1134b, 0x1134d, 0x11350, 0x11350, 0x11357, 0x11357,
|
||||
0x1135d, 0x11363, 0x11366, 0x1136c, 0x11370, 0x11374, 0x11380, 0x11389, 0x1138b, 0x1138b, 0x1138e, 0x1138e,
|
||||
0x11390, 0x113b5, 0x113b7, 0x113c0, 0x113c2, 0x113c2, 0x113c5, 0x113c5, 0x113c7, 0x113ca, 0x113cc, 0x113d3,
|
||||
0x113e1, 0x113e2, 0x11400, 0x1144a, 0x11450, 0x11459, 0x1145e, 0x11461, 0x11480, 0x114c5, 0x114c7, 0x114c7,
|
||||
0x114d0, 0x114d9, 0x11580, 0x115b5, 0x115b8, 0x115c0, 0x115d8, 0x115dd, 0x11600, 0x11640, 0x11644, 0x11644,
|
||||
0x11650, 0x11659, 0x11680, 0x116b8, 0x116c0, 0x116c9, 0x116d0, 0x116e3, 0x11700, 0x1171a, 0x1171d, 0x1172b,
|
||||
0x11730, 0x11739, 0x11740, 0x11746, 0x11800, 0x1183a, 0x118a0, 0x118e9, 0x118ff, 0x11906, 0x11909, 0x11909,
|
||||
0x1190c, 0x11913, 0x11915, 0x11916, 0x11918, 0x11935, 0x11937, 0x11938, 0x1193b, 0x11943, 0x11950, 0x11959,
|
||||
0x119a0, 0x119a7, 0x119aa, 0x119d7, 0x119da, 0x119e1, 0x119e3, 0x119e4, 0x11a00, 0x11a3e, 0x11a47, 0x11a47,
|
||||
0x11a50, 0x11a99, 0x11a9d, 0x11a9d, 0x11ab0, 0x11af8, 0x11b60, 0x11b67, 0x11bc0, 0x11be0, 0x11bf0, 0x11bf9,
|
||||
0x11c00, 0x11c08, 0x11c0a, 0x11c36, 0x11c38, 0x11c40, 0x11c50, 0x11c59, 0x11c72, 0x11c8f, 0x11c92, 0x11ca7,
|
||||
0x11ca9, 0x11cb6, 0x11d00, 0x11d06, 0x11d08, 0x11d09, 0x11d0b, 0x11d36, 0x11d3a, 0x11d3a, 0x11d3c, 0x11d3d,
|
||||
0x11d3f, 0x11d47, 0x11d50, 0x11d59, 0x11d60, 0x11d65, 0x11d67, 0x11d68, 0x11d6a, 0x11d8e, 0x11d90, 0x11d91,
|
||||
0x11d93, 0x11d98, 0x11da0, 0x11da9, 0x11db0, 0x11ddb, 0x11de0, 0x11de9, 0x11ee0, 0x11ef6, 0x11f00, 0x11f10,
|
||||
0x11f12, 0x11f3a, 0x11f3e, 0x11f42, 0x11f50, 0x11f5a, 0x11fb0, 0x11fb0, 0x12000, 0x12399, 0x12400, 0x1246e,
|
||||
0x12480, 0x12543, 0x12f90, 0x12ff0, 0x13000, 0x1342f, 0x13440, 0x13455, 0x13460, 0x143fa, 0x14400, 0x14646,
|
||||
0x16100, 0x16139, 0x16800, 0x16a38, 0x16a40, 0x16a5e, 0x16a60, 0x16a69, 0x16a70, 0x16abe, 0x16ac0, 0x16ac9,
|
||||
0x16ad0, 0x16aed, 0x16af0, 0x16af4, 0x16b00, 0x16b36, 0x16b40, 0x16b43, 0x16b50, 0x16b59, 0x16b63, 0x16b77,
|
||||
0x16b7d, 0x16b8f, 0x16d40, 0x16d6c, 0x16d70, 0x16d79, 0x16e40, 0x16e7f, 0x16ea0, 0x16eb8, 0x16ebb, 0x16ed3,
|
||||
0x16f00, 0x16f4a, 0x16f4f, 0x16f87, 0x16f8f, 0x16f9f, 0x16fe0, 0x16fe1, 0x16fe3, 0x16fe4, 0x16ff0, 0x16ff6,
|
||||
0x17000, 0x18cd5, 0x18cff, 0x18d1e, 0x18d80, 0x18df2, 0x1aff0, 0x1aff3, 0x1aff5, 0x1affb, 0x1affd, 0x1affe,
|
||||
0x1b000, 0x1b122, 0x1b132, 0x1b132, 0x1b150, 0x1b152, 0x1b155, 0x1b155, 0x1b164, 0x1b167, 0x1b170, 0x1b2fb,
|
||||
0x1bc00, 0x1bc6a, 0x1bc70, 0x1bc7c, 0x1bc80, 0x1bc88, 0x1bc90, 0x1bc99, 0x1bc9d, 0x1bc9e, 0x1ccf0, 0x1ccf9,
|
||||
0x1cf00, 0x1cf2d, 0x1cf30, 0x1cf46, 0x1d165, 0x1d169, 0x1d16d, 0x1d172, 0x1d17b, 0x1d182, 0x1d185, 0x1d18b,
|
||||
0x1d1aa, 0x1d1ad, 0x1d242, 0x1d244, 0x1d400, 0x1d454, 0x1d456, 0x1d49c, 0x1d49e, 0x1d49f, 0x1d4a2, 0x1d4a2,
|
||||
0x1d4a5, 0x1d4a6, 0x1d4a9, 0x1d4ac, 0x1d4ae, 0x1d4b9, 0x1d4bb, 0x1d4bb, 0x1d4bd, 0x1d4c3, 0x1d4c5, 0x1d505,
|
||||
0x1d507, 0x1d50a, 0x1d50d, 0x1d514, 0x1d516, 0x1d51c, 0x1d51e, 0x1d539, 0x1d53b, 0x1d53e, 0x1d540, 0x1d544,
|
||||
0x1d546, 0x1d546, 0x1d54a, 0x1d550, 0x1d552, 0x1d6a5, 0x1d6a8, 0x1d6c0, 0x1d6c2, 0x1d6da, 0x1d6dc, 0x1d6fa,
|
||||
0x1d6fc, 0x1d714, 0x1d716, 0x1d734, 0x1d736, 0x1d74e, 0x1d750, 0x1d76e, 0x1d770, 0x1d788, 0x1d78a, 0x1d7a8,
|
||||
0x1d7aa, 0x1d7c2, 0x1d7c4, 0x1d7cb, 0x1d7ce, 0x1d7ff, 0x1da00, 0x1da36, 0x1da3b, 0x1da6c, 0x1da75, 0x1da75,
|
||||
0x1da84, 0x1da84, 0x1da9b, 0x1da9f, 0x1daa1, 0x1daaf, 0x1df00, 0x1df1e, 0x1df25, 0x1df2a, 0x1e000, 0x1e006,
|
||||
0x1e008, 0x1e018, 0x1e01b, 0x1e021, 0x1e023, 0x1e024, 0x1e026, 0x1e02a, 0x1e030, 0x1e06d, 0x1e08f, 0x1e08f,
|
||||
0x1e100, 0x1e12c, 0x1e130, 0x1e13d, 0x1e140, 0x1e149, 0x1e14e, 0x1e14e, 0x1e290, 0x1e2ae, 0x1e2c0, 0x1e2f9,
|
||||
0x1e4d0, 0x1e4f9, 0x1e5d0, 0x1e5fa, 0x1e6c0, 0x1e6de, 0x1e6e0, 0x1e6f5, 0x1e6fe, 0x1e6ff, 0x1e7e0, 0x1e7e6,
|
||||
0x1e7e8, 0x1e7eb, 0x1e7ed, 0x1e7ee, 0x1e7f0, 0x1e7fe, 0x1e800, 0x1e8c4, 0x1e8d0, 0x1e8d6, 0x1e900, 0x1e94b,
|
||||
0x1e950, 0x1e959, 0x1ee00, 0x1ee03, 0x1ee05, 0x1ee1f, 0x1ee21, 0x1ee22, 0x1ee24, 0x1ee24, 0x1ee27, 0x1ee27,
|
||||
0x1ee29, 0x1ee32, 0x1ee34, 0x1ee37, 0x1ee39, 0x1ee39, 0x1ee3b, 0x1ee3b, 0x1ee42, 0x1ee42, 0x1ee47, 0x1ee47,
|
||||
0x1ee49, 0x1ee49, 0x1ee4b, 0x1ee4b, 0x1ee4d, 0x1ee4f, 0x1ee51, 0x1ee52, 0x1ee54, 0x1ee54, 0x1ee57, 0x1ee57,
|
||||
0x1ee59, 0x1ee59, 0x1ee5b, 0x1ee5b, 0x1ee5d, 0x1ee5d, 0x1ee5f, 0x1ee5f, 0x1ee61, 0x1ee62, 0x1ee64, 0x1ee64,
|
||||
0x1ee67, 0x1ee6a, 0x1ee6c, 0x1ee72, 0x1ee74, 0x1ee77, 0x1ee79, 0x1ee7c, 0x1ee7e, 0x1ee7e, 0x1ee80, 0x1ee89,
|
||||
0x1ee8b, 0x1ee9b, 0x1eea1, 0x1eea3, 0x1eea5, 0x1eea9, 0x1eeab, 0x1eebb, 0x1fbf0, 0x1fbf9, 0x20000, 0x2a6df,
|
||||
0x2a700, 0x2b81d, 0x2b820, 0x2cead, 0x2ceb0, 0x2ebe0, 0x2ebf0, 0x2ee5d, 0x2f800, 0x2fa1d, 0x30000, 0x3134a,
|
||||
0x31350, 0x33479, 0xe0100, 0xe01ef,
|
||||
];
|
||||
|
||||
export const DEFAULT_IGNORABLE_RANGES: readonly number[] = [
|
||||
0xad, 0xad, 0x34f, 0x34f, 0x61c, 0x61c, 0x115f, 0x1160, 0x17b4, 0x17b5, 0x180b, 0x180f,
|
||||
0x200b, 0x200f, 0x202a, 0x202e, 0x2060, 0x206f, 0x3164, 0x3164, 0xfe00, 0xfe0f, 0xfeff, 0xfeff,
|
||||
0xffa0, 0xffa0, 0xfff0, 0xfff8, 0x1bca0, 0x1bca3, 0x1d173, 0x1d17a, 0xe0000, 0xe0fff,
|
||||
];
|
||||
|
||||
export const COMBINING_MARK_RANGES: readonly number[] = [
|
||||
0x300, 0x36f, 0x483, 0x487, 0x591, 0x5bd, 0x5bf, 0x5bf, 0x5c1, 0x5c2, 0x5c4, 0x5c5,
|
||||
0x5c7, 0x5c7, 0x610, 0x61a, 0x64b, 0x65f, 0x670, 0x670, 0x6d6, 0x6dc, 0x6df, 0x6e4,
|
||||
0x6e7, 0x6e8, 0x6ea, 0x6ed, 0x711, 0x711, 0x730, 0x74a, 0x7a6, 0x7b0, 0x7eb, 0x7f3,
|
||||
0x7fd, 0x7fd, 0x816, 0x819, 0x81b, 0x823, 0x825, 0x827, 0x829, 0x82d, 0x859, 0x85b,
|
||||
0x897, 0x89f, 0x8ca, 0x8e1, 0x8e3, 0x903, 0x93a, 0x93c, 0x93e, 0x94f, 0x951, 0x957,
|
||||
0x962, 0x963, 0x981, 0x983, 0x9bc, 0x9bc, 0x9be, 0x9c4, 0x9c7, 0x9c8, 0x9cb, 0x9cd,
|
||||
0x9d7, 0x9d7, 0x9e2, 0x9e3, 0x9fe, 0x9fe, 0xa01, 0xa03, 0xa3c, 0xa3c, 0xa3e, 0xa42,
|
||||
0xa47, 0xa48, 0xa4b, 0xa4d, 0xa51, 0xa51, 0xa70, 0xa71, 0xa75, 0xa75, 0xa81, 0xa83,
|
||||
0xabc, 0xabc, 0xabe, 0xac5, 0xac7, 0xac9, 0xacb, 0xacd, 0xae2, 0xae3, 0xafa, 0xaff,
|
||||
0xb01, 0xb03, 0xb3c, 0xb3c, 0xb3e, 0xb44, 0xb47, 0xb48, 0xb4b, 0xb4d, 0xb55, 0xb57,
|
||||
0xb62, 0xb63, 0xb82, 0xb82, 0xbbe, 0xbc2, 0xbc6, 0xbc8, 0xbca, 0xbcd, 0xbd7, 0xbd7,
|
||||
0xc00, 0xc04, 0xc3c, 0xc3c, 0xc3e, 0xc44, 0xc46, 0xc48, 0xc4a, 0xc4d, 0xc55, 0xc56,
|
||||
0xc62, 0xc63, 0xc81, 0xc83, 0xcbc, 0xcbc, 0xcbe, 0xcc4, 0xcc6, 0xcc8, 0xcca, 0xccd,
|
||||
0xcd5, 0xcd6, 0xce2, 0xce3, 0xcf3, 0xcf3, 0xd00, 0xd03, 0xd3b, 0xd3c, 0xd3e, 0xd44,
|
||||
0xd46, 0xd48, 0xd4a, 0xd4d, 0xd57, 0xd57, 0xd62, 0xd63, 0xd81, 0xd83, 0xdca, 0xdca,
|
||||
0xdcf, 0xdd4, 0xdd6, 0xdd6, 0xdd8, 0xddf, 0xdf2, 0xdf3, 0xe31, 0xe31, 0xe34, 0xe3a,
|
||||
0xe47, 0xe4e, 0xeb1, 0xeb1, 0xeb4, 0xebc, 0xec8, 0xece, 0xf18, 0xf19, 0xf35, 0xf35,
|
||||
0xf37, 0xf37, 0xf39, 0xf39, 0xf3e, 0xf3f, 0xf71, 0xf84, 0xf86, 0xf87, 0xf8d, 0xf97,
|
||||
0xf99, 0xfbc, 0xfc6, 0xfc6, 0x102b, 0x103e, 0x1056, 0x1059, 0x105e, 0x1060, 0x1062, 0x1064,
|
||||
0x1067, 0x106d, 0x1071, 0x1074, 0x1082, 0x108d, 0x108f, 0x108f, 0x109a, 0x109d, 0x135d, 0x135f,
|
||||
0x1712, 0x1715, 0x1732, 0x1734, 0x1752, 0x1753, 0x1772, 0x1773, 0x17b4, 0x17d3, 0x17dd, 0x17dd,
|
||||
0x180b, 0x180d, 0x180f, 0x180f, 0x1885, 0x1886, 0x18a9, 0x18a9, 0x1920, 0x192b, 0x1930, 0x193b,
|
||||
0x1a17, 0x1a1b, 0x1a55, 0x1a5e, 0x1a60, 0x1a7c, 0x1a7f, 0x1a7f, 0x1ab0, 0x1abd, 0x1abf, 0x1add,
|
||||
0x1ae0, 0x1aeb, 0x1b00, 0x1b04, 0x1b34, 0x1b44, 0x1b6b, 0x1b73, 0x1b80, 0x1b82, 0x1ba1, 0x1bad,
|
||||
0x1be6, 0x1bf3, 0x1c24, 0x1c37, 0x1cd0, 0x1cd2, 0x1cd4, 0x1ce8, 0x1ced, 0x1ced, 0x1cf4, 0x1cf4,
|
||||
0x1cf7, 0x1cf9, 0x1dc0, 0x1dff, 0x20d0, 0x20dc, 0x20e1, 0x20e1, 0x20e5, 0x20f0, 0x2cef, 0x2cf1,
|
||||
0x2d7f, 0x2d7f, 0x2de0, 0x2dff, 0x302a, 0x302f, 0x3099, 0x309a, 0xa66f, 0xa66f, 0xa674, 0xa67d,
|
||||
0xa69e, 0xa69f, 0xa6f0, 0xa6f1, 0xa802, 0xa802, 0xa806, 0xa806, 0xa80b, 0xa80b, 0xa823, 0xa827,
|
||||
0xa82c, 0xa82c, 0xa880, 0xa881, 0xa8b4, 0xa8c5, 0xa8e0, 0xa8f1, 0xa8ff, 0xa8ff, 0xa926, 0xa92d,
|
||||
0xa947, 0xa953, 0xa980, 0xa983, 0xa9b3, 0xa9c0, 0xa9e5, 0xa9e5, 0xaa29, 0xaa36, 0xaa43, 0xaa43,
|
||||
0xaa4c, 0xaa4d, 0xaa7b, 0xaa7d, 0xaab0, 0xaab0, 0xaab2, 0xaab4, 0xaab7, 0xaab8, 0xaabe, 0xaabf,
|
||||
0xaac1, 0xaac1, 0xaaeb, 0xaaef, 0xaaf5, 0xaaf6, 0xabe3, 0xabea, 0xabec, 0xabed, 0xfb1e, 0xfb1e,
|
||||
0xfe00, 0xfe0f, 0xfe20, 0xfe2f, 0x101fd, 0x101fd, 0x102e0, 0x102e0, 0x10376, 0x1037a, 0x10a01, 0x10a03,
|
||||
0x10a05, 0x10a06, 0x10a0c, 0x10a0f, 0x10a38, 0x10a3a, 0x10a3f, 0x10a3f, 0x10ae5, 0x10ae6, 0x10d24, 0x10d27,
|
||||
0x10d69, 0x10d6d, 0x10eab, 0x10eac, 0x10efa, 0x10eff, 0x10f46, 0x10f50, 0x10f82, 0x10f85, 0x11000, 0x11002,
|
||||
0x11038, 0x11046, 0x11070, 0x11070, 0x11073, 0x11074, 0x1107f, 0x11082, 0x110b0, 0x110ba, 0x110c2, 0x110c2,
|
||||
0x11100, 0x11102, 0x11127, 0x11134, 0x11145, 0x11146, 0x11173, 0x11173, 0x11180, 0x11182, 0x111b3, 0x111c0,
|
||||
0x111c9, 0x111cc, 0x111ce, 0x111cf, 0x1122c, 0x11237, 0x1123e, 0x1123e, 0x11241, 0x11241, 0x112df, 0x112ea,
|
||||
0x11300, 0x11303, 0x1133b, 0x1133c, 0x1133e, 0x11344, 0x11347, 0x11348, 0x1134b, 0x1134d, 0x11357, 0x11357,
|
||||
0x11362, 0x11363, 0x11366, 0x1136c, 0x11370, 0x11374, 0x113b8, 0x113c0, 0x113c2, 0x113c2, 0x113c5, 0x113c5,
|
||||
0x113c7, 0x113ca, 0x113cc, 0x113d0, 0x113d2, 0x113d2, 0x113e1, 0x113e2, 0x11435, 0x11446, 0x1145e, 0x1145e,
|
||||
0x114b0, 0x114c3, 0x115af, 0x115b5, 0x115b8, 0x115c0, 0x115dc, 0x115dd, 0x11630, 0x11640, 0x116ab, 0x116b7,
|
||||
0x1171d, 0x1172b, 0x1182c, 0x1183a, 0x11930, 0x11935, 0x11937, 0x11938, 0x1193b, 0x1193e, 0x11940, 0x11940,
|
||||
0x11942, 0x11943, 0x119d1, 0x119d7, 0x119da, 0x119e0, 0x119e4, 0x119e4, 0x11a01, 0x11a0a, 0x11a33, 0x11a39,
|
||||
0x11a3b, 0x11a3e, 0x11a47, 0x11a47, 0x11a51, 0x11a5b, 0x11a8a, 0x11a99, 0x11b60, 0x11b67, 0x11c2f, 0x11c36,
|
||||
0x11c38, 0x11c3f, 0x11c92, 0x11ca7, 0x11ca9, 0x11cb6, 0x11d31, 0x11d36, 0x11d3a, 0x11d3a, 0x11d3c, 0x11d3d,
|
||||
0x11d3f, 0x11d45, 0x11d47, 0x11d47, 0x11d8a, 0x11d8e, 0x11d90, 0x11d91, 0x11d93, 0x11d97, 0x11ef3, 0x11ef6,
|
||||
0x11f00, 0x11f01, 0x11f03, 0x11f03, 0x11f34, 0x11f3a, 0x11f3e, 0x11f42, 0x11f5a, 0x11f5a, 0x13440, 0x13440,
|
||||
0x13447, 0x13455, 0x1611e, 0x1612f, 0x16af0, 0x16af4, 0x16b30, 0x16b36, 0x16f4f, 0x16f4f, 0x16f51, 0x16f87,
|
||||
0x16f8f, 0x16f92, 0x16fe4, 0x16fe4, 0x16ff0, 0x16ff1, 0x1bc9d, 0x1bc9e, 0x1cf00, 0x1cf2d, 0x1cf30, 0x1cf46,
|
||||
0x1d165, 0x1d169, 0x1d16d, 0x1d172, 0x1d17b, 0x1d182, 0x1d185, 0x1d18b, 0x1d1aa, 0x1d1ad, 0x1d242, 0x1d244,
|
||||
0x1da00, 0x1da36, 0x1da3b, 0x1da6c, 0x1da75, 0x1da75, 0x1da84, 0x1da84, 0x1da9b, 0x1da9f, 0x1daa1, 0x1daaf,
|
||||
0x1e000, 0x1e006, 0x1e008, 0x1e018, 0x1e01b, 0x1e021, 0x1e023, 0x1e024, 0x1e026, 0x1e02a, 0x1e08f, 0x1e08f,
|
||||
0x1e130, 0x1e136, 0x1e2ae, 0x1e2ae, 0x1e2ec, 0x1e2ef, 0x1e4ec, 0x1e4ef, 0x1e5ee, 0x1e5ef, 0x1e6e3, 0x1e6e3,
|
||||
0x1e6e6, 0x1e6e6, 0x1e6ee, 0x1e6ef, 0x1e6f5, 0x1e6f5, 0x1e8d0, 0x1e8d6, 0x1e944, 0x1e94a, 0xe0100, 0xe01ef,
|
||||
];
|
||||
|
||||
export const FULLY_QUALIFIED_EMOJI: readonly string[] = [
|
||||
"😀", "😃", "😄", "😁", "😆", "😅", "🤣", "😂", "🙂", "🙃", "🫠", "😉",
|
||||
"😊", "😇", "🥰", "😍", "🤩", "😘", "😗", "☺️", "😚", "😙", "🥲", "😋",
|
||||
"😛", "😜", "🤪", "😝", "🤑", "🤗", "🤭", "🫢", "🫣", "🤫", "🤔", "🫡",
|
||||
"🤐", "🤨", "😐", "😑", "😶", "🫥", "😶🌫️", "😏", "😒", "🙄", "😬", "😮💨",
|
||||
"🤥", "🫨", "🙂↔️", "🙂↕️", "😌", "😔", "😪", "🤤", "😴", "", "😷", "🤒",
|
||||
"🤕", "🤢", "🤮", "🤧", "🥵", "🥶", "🥴", "😵", "😵💫", "🤯", "🤠", "🥳",
|
||||
"🥸", "😎", "🤓", "🧐", "😕", "🫤", "😟", "🙁", "☹️", "😮", "😯", "😲",
|
||||
"😳", "", "🥺", "🥹", "😦", "😧", "😨", "😰", "😥", "😢", "😭", "😱",
|
||||
"😖", "😣", "😞", "😓", "😩", "😫", "🥱", "😤", "😡", "😠", "🤬", "😈",
|
||||
"👿", "💀", "☠️", "💩", "🤡", "👹", "👺", "👻", "👽", "👾", "🤖", "😺",
|
||||
"😸", "😹", "😻", "😼", "😽", "🙀", "😿", "😾", "🙈", "🙉", "🙊", "💌",
|
||||
"💘", "💝", "💖", "💗", "💓", "💞", "💕", "💟", "❣️", "💔", "❤️🔥", "❤️🩹",
|
||||
"❤️", "🩷", "🧡", "💛", "💚", "💙", "🩵", "💜", "🤎", "🖤", "🩶", "🤍",
|
||||
"💋", "💯", "💢", "", "💥", "💫", "💦", "💨", "🕳️", "💬", "👁️🗨️", "🗨️",
|
||||
"🗯️", "💭", "💤", "👋", "👋🏻", "👋🏼", "👋🏽", "👋🏾", "👋🏿", "🤚", "🤚🏻", "🤚🏼",
|
||||
"🤚🏽", "🤚🏾", "🤚🏿", "🖐️", "🖐🏻", "🖐🏼", "🖐🏽", "🖐🏾", "🖐🏿", "✋", "✋🏻", "✋🏼",
|
||||
"✋🏽", "✋🏾", "✋🏿", "🖖", "🖖🏻", "🖖🏼", "🖖🏽", "🖖🏾", "🖖🏿", "🫱", "🫱🏻", "🫱🏼",
|
||||
"🫱🏽", "🫱🏾", "🫱🏿", "🫲", "🫲🏻", "🫲🏼", "🫲🏽", "🫲🏾", "🫲🏿", "🫳", "🫳🏻", "🫳🏼",
|
||||
"🫳🏽", "🫳🏾", "🫳🏿", "🫴", "🫴🏻", "🫴🏼", "🫴🏽", "🫴🏾", "🫴🏿", "🫷", "🫷🏻", "🫷🏼",
|
||||
"🫷🏽", "🫷🏾", "🫷🏿", "🫸", "🫸🏻", "🫸🏼", "🫸🏽", "🫸🏾", "🫸🏿", "👌", "👌🏻", "👌🏼",
|
||||
"👌🏽", "👌🏾", "👌🏿", "🤌", "🤌🏻", "🤌🏼", "🤌🏽", "🤌🏾", "🤌🏿", "🤏", "🤏🏻", "🤏🏼",
|
||||
"🤏🏽", "🤏🏾", "🤏🏿", "✌️", "✌🏻", "✌🏼", "✌🏽", "✌🏾", "✌🏿", "🤞", "🤞🏻", "🤞🏼",
|
||||
"🤞🏽", "🤞🏾", "🤞🏿", "🫰", "🫰🏻", "🫰🏼", "🫰🏽", "🫰🏾", "🫰🏿", "🤟", "🤟🏻", "🤟🏼",
|
||||
"🤟🏽", "🤟🏾", "🤟🏿", "🤘", "🤘🏻", "🤘🏼", "🤘🏽", "🤘🏾", "🤘🏿", "🤙", "🤙🏻", "🤙🏼",
|
||||
"🤙🏽", "🤙🏾", "🤙🏿", "👈", "👈🏻", "👈🏼", "👈🏽", "👈🏾", "👈🏿", "👉", "👉🏻", "👉🏼",
|
||||
"👉🏽", "👉🏾", "👉🏿", "👆", "👆🏻", "👆🏼", "👆🏽", "👆🏾", "👆🏿", "🖕", "🖕🏻", "🖕🏼",
|
||||
"🖕🏽", "🖕🏾", "🖕🏿", "👇", "👇🏻", "👇🏼", "👇🏽", "👇🏾", "👇🏿", "☝️", "☝🏻", "☝🏼",
|
||||
"☝🏽", "☝🏾", "☝🏿", "🫵", "🫵🏻", "🫵🏼", "🫵🏽", "🫵🏾", "🫵🏿", "👍", "👍🏻", "👍🏼",
|
||||
"👍🏽", "👍🏾", "👍🏿", "👎", "👎🏻", "👎🏼", "👎🏽", "👎🏾", "👎🏿", "✊", "✊🏻", "✊🏼",
|
||||
"✊🏽", "✊🏾", "✊🏿", "👊", "👊🏻", "👊🏼", "👊🏽", "👊🏾", "👊🏿", "🤛", "🤛🏻", "🤛🏼",
|
||||
"🤛🏽", "🤛🏾", "🤛🏿", "🤜", "🤜🏻", "🤜🏼", "🤜🏽", "🤜🏾", "🤜🏿", "👏", "👏🏻", "👏🏼",
|
||||
"👏🏽", "👏🏾", "👏🏿", "🙌", "🙌🏻", "🙌🏼", "🙌🏽", "🙌🏾", "🙌🏿", "🫶", "🫶🏻", "🫶🏼",
|
||||
"🫶🏽", "🫶🏾", "🫶🏿", "👐", "👐🏻", "👐🏼", "👐🏽", "👐🏾", "👐🏿", "🤲", "🤲🏻", "🤲🏼",
|
||||
"🤲🏽", "🤲🏾", "🤲🏿", "🤝", "🤝🏻", "🤝🏼", "🤝🏽", "🤝🏾", "🤝🏿", "🫱🏻🫲🏼", "🫱🏻🫲🏽", "🫱🏻🫲🏾",
|
||||
"🫱🏻🫲🏿", "🫱🏼🫲🏻", "🫱🏼🫲🏽", "🫱🏼🫲🏾", "🫱🏼🫲🏿", "🫱🏽🫲🏻", "🫱🏽🫲🏼", "🫱🏽🫲🏾", "🫱🏽🫲🏿", "🫱🏾🫲🏻", "🫱🏾🫲🏼", "🫱🏾🫲🏽",
|
||||
"🫱🏾🫲🏿", "🫱🏿🫲🏻", "🫱🏿🫲🏼", "🫱🏿🫲🏽", "🫱🏿🫲🏾", "🙏", "🙏🏻", "🙏🏼", "🙏🏽", "🙏🏾", "🙏🏿", "✍️",
|
||||
"✍🏻", "✍🏼", "✍🏽", "✍🏾", "✍🏿", "💅", "💅🏻", "💅🏼", "💅🏽", "💅🏾", "💅🏿", "🤳",
|
||||
"🤳🏻", "🤳🏼", "🤳🏽", "🤳🏾", "🤳🏿", "💪", "💪🏻", "💪🏼", "💪🏽", "💪🏾", "💪🏿", "🦾",
|
||||
"🦿", "🦵", "🦵🏻", "🦵🏼", "🦵🏽", "🦵🏾", "🦵🏿", "🦶", "🦶🏻", "🦶🏼", "🦶🏽", "🦶🏾",
|
||||
"🦶🏿", "👂", "👂🏻", "👂🏼", "👂🏽", "👂🏾", "👂🏿", "🦻", "🦻🏻", "🦻🏼", "🦻🏽", "🦻🏾",
|
||||
"🦻🏿", "👃", "👃🏻", "👃🏼", "👃🏽", "👃🏾", "👃🏿", "🧠", "🫀", "🫁", "🦷", "🦴",
|
||||
"👀", "👁️", "👅", "👄", "🫦", "👶", "👶🏻", "👶🏼", "👶🏽", "👶🏾", "👶🏿", "🧒",
|
||||
"🧒🏻", "🧒🏼", "🧒🏽", "🧒🏾", "🧒🏿", "👦", "👦🏻", "👦🏼", "👦🏽", "👦🏾", "👦🏿", "👧",
|
||||
"👧🏻", "👧🏼", "👧🏽", "👧🏾", "👧🏿", "🧑", "🧑🏻", "🧑🏼", "🧑🏽", "🧑🏾", "🧑🏿", "👱",
|
||||
"👱🏻", "👱🏼", "👱🏽", "👱🏾", "👱🏿", "👨", "👨🏻", "👨🏼", "👨🏽", "👨🏾", "👨🏿", "🧔",
|
||||
"🧔🏻", "🧔🏼", "🧔🏽", "🧔🏾", "🧔🏿", "🧔♂️", "🧔🏻♂️", "🧔🏼♂️", "🧔🏽♂️", "🧔🏾♂️", "🧔🏿♂️", "🧔♀️",
|
||||
"🧔🏻♀️", "🧔🏼♀️", "🧔🏽♀️", "🧔🏾♀️", "🧔🏿♀️", "👨🦰", "👨🏻🦰", "👨🏼🦰", "👨🏽🦰", "👨🏾🦰", "👨🏿🦰", "👨🦱",
|
||||
"👨🏻🦱", "👨🏼🦱", "👨🏽🦱", "👨🏾🦱", "👨🏿🦱", "👨🦳", "👨🏻🦳", "👨🏼🦳", "👨🏽🦳", "👨🏾🦳", "👨🏿🦳", "👨🦲",
|
||||
"👨🏻🦲", "👨🏼🦲", "👨🏽🦲", "👨🏾🦲", "👨🏿🦲", "👩", "👩🏻", "👩🏼", "👩🏽", "👩🏾", "👩🏿", "👩🦰",
|
||||
"👩🏻🦰", "👩🏼🦰", "👩🏽🦰", "👩🏾🦰", "👩🏿🦰", "🧑🦰", "🧑🏻🦰", "🧑🏼🦰", "🧑🏽🦰", "🧑🏾🦰", "🧑🏿🦰", "👩🦱",
|
||||
"👩🏻🦱", "👩🏼🦱", "👩🏽🦱", "👩🏾🦱", "👩🏿🦱", "🧑🦱", "🧑🏻🦱", "🧑🏼🦱", "🧑🏽🦱", "🧑🏾🦱", "🧑🏿🦱", "👩🦳",
|
||||
"👩🏻🦳", "👩🏼🦳", "👩🏽🦳", "👩🏾🦳", "👩🏿🦳", "🧑🦳", "🧑🏻🦳", "🧑🏼🦳", "🧑🏽🦳", "🧑🏾🦳", "🧑🏿🦳", "👩🦲",
|
||||
"👩🏻🦲", "👩🏼🦲", "👩🏽🦲", "👩🏾🦲", "👩🏿🦲", "🧑🦲", "🧑🏻🦲", "🧑🏼🦲", "🧑🏽🦲", "🧑🏾🦲", "🧑🏿🦲", "👱♀️",
|
||||
"👱🏻♀️", "👱🏼♀️", "👱🏽♀️", "👱🏾♀️", "👱🏿♀️", "👱♂️", "👱🏻♂️", "👱🏼♂️", "👱🏽♂️", "👱🏾♂️", "👱🏿♂️", "🧓",
|
||||
"🧓🏻", "🧓🏼", "🧓🏽", "🧓🏾", "🧓🏿", "👴", "👴🏻", "👴🏼", "👴🏽", "👴🏾", "👴🏿", "👵",
|
||||
"👵🏻", "👵🏼", "👵🏽", "👵🏾", "👵🏿", "🙍", "🙍🏻", "🙍🏼", "🙍🏽", "🙍🏾", "🙍🏿", "🙍♂️",
|
||||
"🙍🏻♂️", "🙍🏼♂️", "🙍🏽♂️", "🙍🏾♂️", "🙍🏿♂️", "🙍♀️", "🙍🏻♀️", "🙍🏼♀️", "🙍🏽♀️", "🙍🏾♀️", "🙍🏿♀️", "🙎",
|
||||
"🙎🏻", "🙎🏼", "🙎🏽", "🙎🏾", "🙎🏿", "🙎♂️", "🙎🏻♂️", "🙎🏼♂️", "🙎🏽♂️", "🙎🏾♂️", "🙎🏿♂️", "🙎♀️",
|
||||
"🙎🏻♀️", "🙎🏼♀️", "🙎🏽♀️", "🙎🏾♀️", "🙎🏿♀️", "🙅", "🙅🏻", "🙅🏼", "🙅🏽", "🙅🏾", "🙅🏿", "🙅♂️",
|
||||
"🙅🏻♂️", "🙅🏼♂️", "🙅🏽♂️", "🙅🏾♂️", "🙅🏿♂️", "🙅♀️", "🙅🏻♀️", "🙅🏼♀️", "🙅🏽♀️", "🙅🏾♀️", "🙅🏿♀️", "🙆",
|
||||
"🙆🏻", "🙆🏼", "🙆🏽", "🙆🏾", "🙆🏿", "🙆♂️", "🙆🏻♂️", "🙆🏼♂️", "🙆🏽♂️", "🙆🏾♂️", "🙆🏿♂️", "🙆♀️",
|
||||
"🙆🏻♀️", "🙆🏼♀️", "🙆🏽♀️", "🙆🏾♀️", "🙆🏿♀️", "💁", "💁🏻", "💁🏼", "💁🏽", "💁🏾", "💁🏿", "💁♂️",
|
||||
"💁🏻♂️", "💁🏼♂️", "💁🏽♂️", "💁🏾♂️", "💁🏿♂️", "💁♀️", "💁🏻♀️", "💁🏼♀️", "💁🏽♀️", "💁🏾♀️", "💁🏿♀️", "🙋",
|
||||
"🙋🏻", "🙋🏼", "🙋🏽", "🙋🏾", "🙋🏿", "🙋♂️", "🙋🏻♂️", "🙋🏼♂️", "🙋🏽♂️", "🙋🏾♂️", "🙋🏿♂️", "🙋♀️",
|
||||
"🙋🏻♀️", "🙋🏼♀️", "🙋🏽♀️", "🙋🏾♀️", "🙋🏿♀️", "🧏", "🧏🏻", "🧏🏼", "🧏🏽", "🧏🏾", "🧏🏿", "🧏♂️",
|
||||
"🧏🏻♂️", "🧏🏼♂️", "🧏🏽♂️", "🧏🏾♂️", "🧏🏿♂️", "🧏♀️", "🧏🏻♀️", "🧏🏼♀️", "🧏🏽♀️", "🧏🏾♀️", "🧏🏿♀️", "🙇",
|
||||
"🙇🏻", "🙇🏼", "🙇🏽", "🙇🏾", "🙇🏿", "🙇♂️", "🙇🏻♂️", "🙇🏼♂️", "🙇🏽♂️", "🙇🏾♂️", "🙇🏿♂️", "🙇♀️",
|
||||
"🙇🏻♀️", "🙇🏼♀️", "🙇🏽♀️", "🙇🏾♀️", "🙇🏿♀️", "🤦", "🤦🏻", "🤦🏼", "🤦🏽", "🤦🏾", "🤦🏿", "🤦♂️",
|
||||
"🤦🏻♂️", "🤦🏼♂️", "🤦🏽♂️", "🤦🏾♂️", "🤦🏿♂️", "🤦♀️", "🤦🏻♀️", "🤦🏼♀️", "🤦🏽♀️", "🤦🏾♀️", "🤦🏿♀️", "🤷",
|
||||
"🤷🏻", "🤷🏼", "🤷🏽", "🤷🏾", "🤷🏿", "🤷♂️", "🤷🏻♂️", "🤷🏼♂️", "🤷🏽♂️", "🤷🏾♂️", "🤷🏿♂️", "🤷♀️",
|
||||
"🤷🏻♀️", "🤷🏼♀️", "🤷🏽♀️", "🤷🏾♀️", "🤷🏿♀️", "🧑⚕️", "🧑🏻⚕️", "🧑🏼⚕️", "🧑🏽⚕️", "🧑🏾⚕️", "🧑🏿⚕️", "👨⚕️",
|
||||
"👨🏻⚕️", "👨🏼⚕️", "👨🏽⚕️", "👨🏾⚕️", "👨🏿⚕️", "👩⚕️", "👩🏻⚕️", "👩🏼⚕️", "👩🏽⚕️", "👩🏾⚕️", "👩🏿⚕️", "🧑🎓",
|
||||
"🧑🏻🎓", "🧑🏼🎓", "🧑🏽🎓", "🧑🏾🎓", "🧑🏿🎓", "👨🎓", "👨🏻🎓", "👨🏼🎓", "👨🏽🎓", "👨🏾🎓", "👨🏿🎓", "👩🎓",
|
||||
"👩🏻🎓", "👩🏼🎓", "👩🏽🎓", "👩🏾🎓", "👩🏿🎓", "🧑🏫", "🧑🏻🏫", "🧑🏼🏫", "🧑🏽🏫", "🧑🏾🏫", "🧑🏿🏫", "👨🏫",
|
||||
"👨🏻🏫", "👨🏼🏫", "👨🏽🏫", "👨🏾🏫", "👨🏿🏫", "👩🏫", "👩🏻🏫", "👩🏼🏫", "👩🏽🏫", "👩🏾🏫", "👩🏿🏫", "🧑⚖️",
|
||||
"🧑🏻⚖️", "🧑🏼⚖️", "🧑🏽⚖️", "🧑🏾⚖️", "🧑🏿⚖️", "👨⚖️", "👨🏻⚖️", "👨🏼⚖️", "👨🏽⚖️", "👨🏾⚖️", "👨🏿⚖️", "👩⚖️",
|
||||
"👩🏻⚖️", "👩🏼⚖️", "👩🏽⚖️", "👩🏾⚖️", "👩🏿⚖️", "🧑🌾", "🧑🏻🌾", "🧑🏼🌾", "🧑🏽🌾", "🧑🏾🌾", "🧑🏿🌾", "👨🌾",
|
||||
"👨🏻🌾", "👨🏼🌾", "👨🏽🌾", "👨🏾🌾", "👨🏿🌾", "👩🌾", "👩🏻🌾", "👩🏼🌾", "👩🏽🌾", "👩🏾🌾", "👩🏿🌾", "🧑🍳",
|
||||
"🧑🏻🍳", "🧑🏼🍳", "🧑🏽🍳", "🧑🏾🍳", "🧑🏿🍳", "👨🍳", "👨🏻🍳", "👨🏼🍳", "👨🏽🍳", "👨🏾🍳", "👨🏿🍳", "👩🍳",
|
||||
"👩🏻🍳", "👩🏼🍳", "👩🏽🍳", "👩🏾🍳", "👩🏿🍳", "🧑🔧", "🧑🏻🔧", "🧑🏼🔧", "🧑🏽🔧", "🧑🏾🔧", "🧑🏿🔧", "👨🔧",
|
||||
"👨🏻🔧", "👨🏼🔧", "👨🏽🔧", "👨🏾🔧", "👨🏿🔧", "👩🔧", "👩🏻🔧", "👩🏼🔧", "👩🏽🔧", "👩🏾🔧", "👩🏿🔧", "🧑🏭",
|
||||
"🧑🏻🏭", "🧑🏼🏭", "🧑🏽🏭", "🧑🏾🏭", "🧑🏿🏭", "👨🏭", "👨🏻🏭", "👨🏼🏭", "👨🏽🏭", "👨🏾🏭", "👨🏿🏭", "👩🏭",
|
||||
"👩🏻🏭", "👩🏼🏭", "👩🏽🏭", "👩🏾🏭", "👩🏿🏭", "🧑💼", "🧑🏻💼", "🧑🏼💼", "🧑🏽💼", "🧑🏾💼", "🧑🏿💼", "👨💼",
|
||||
"👨🏻💼", "👨🏼💼", "👨🏽💼", "👨🏾💼", "👨🏿💼", "👩💼", "👩🏻💼", "👩🏼💼", "👩🏽💼", "👩🏾💼", "👩🏿💼", "🧑🔬",
|
||||
"🧑🏻🔬", "🧑🏼🔬", "🧑🏽🔬", "🧑🏾🔬", "🧑🏿🔬", "👨🔬", "👨🏻🔬", "👨🏼🔬", "👨🏽🔬", "👨🏾🔬", "👨🏿🔬", "👩🔬",
|
||||
"👩🏻🔬", "👩🏼🔬", "👩🏽🔬", "👩🏾🔬", "👩🏿🔬", "🧑💻", "🧑🏻💻", "🧑🏼💻", "🧑🏽💻", "🧑🏾💻", "🧑🏿💻", "👨💻",
|
||||
"👨🏻💻", "👨🏼💻", "👨🏽💻", "👨🏾💻", "👨🏿💻", "👩💻", "👩🏻💻", "👩🏼💻", "👩🏽💻", "👩🏾💻", "👩🏿💻", "🧑🎤",
|
||||
"🧑🏻🎤", "🧑🏼🎤", "🧑🏽🎤", "🧑🏾🎤", "🧑🏿🎤", "👨🎤", "👨🏻🎤", "👨🏼🎤", "👨🏽🎤", "👨🏾🎤", "👨🏿🎤", "👩🎤",
|
||||
"👩🏻🎤", "👩🏼🎤", "👩🏽🎤", "👩🏾🎤", "👩🏿🎤", "🧑🎨", "🧑🏻🎨", "🧑🏼🎨", "🧑🏽🎨", "🧑🏾🎨", "🧑🏿🎨", "👨🎨",
|
||||
"👨🏻🎨", "👨🏼🎨", "👨🏽🎨", "👨🏾🎨", "👨🏿🎨", "👩🎨", "👩🏻🎨", "👩🏼🎨", "👩🏽🎨", "👩🏾🎨", "👩🏿🎨", "🧑✈️",
|
||||
"🧑🏻✈️", "🧑🏼✈️", "🧑🏽✈️", "🧑🏾✈️", "🧑🏿✈️", "👨✈️", "👨🏻✈️", "👨🏼✈️", "👨🏽✈️", "👨🏾✈️", "👨🏿✈️", "👩✈️",
|
||||
"👩🏻✈️", "👩🏼✈️", "👩🏽✈️", "👩🏾✈️", "👩🏿✈️", "🧑🚀", "🧑🏻🚀", "🧑🏼🚀", "🧑🏽🚀", "🧑🏾🚀", "🧑🏿🚀", "👨🚀",
|
||||
"👨🏻🚀", "👨🏼🚀", "👨🏽🚀", "👨🏾🚀", "👨🏿🚀", "👩🚀", "👩🏻🚀", "👩🏼🚀", "👩🏽🚀", "👩🏾🚀", "👩🏿🚀", "🧑🚒",
|
||||
"🧑🏻🚒", "🧑🏼🚒", "🧑🏽🚒", "🧑🏾🚒", "🧑🏿🚒", "👨🚒", "👨🏻🚒", "👨🏼🚒", "👨🏽🚒", "👨🏾🚒", "👨🏿🚒", "👩🚒",
|
||||
"👩🏻🚒", "👩🏼🚒", "👩🏽🚒", "👩🏾🚒", "👩🏿🚒", "👮", "👮🏻", "👮🏼", "👮🏽", "👮🏾", "👮🏿", "👮♂️",
|
||||
"👮🏻♂️", "👮🏼♂️", "👮🏽♂️", "👮🏾♂️", "👮🏿♂️", "👮♀️", "👮🏻♀️", "👮🏼♀️", "👮🏽♀️", "👮🏾♀️", "👮🏿♀️", "🕵️",
|
||||
"🕵🏻", "🕵🏼", "🕵🏽", "🕵🏾", "🕵🏿", "🕵️♂️", "🕵🏻♂️", "🕵🏼♂️", "🕵🏽♂️", "🕵🏾♂️", "🕵🏿♂️", "🕵️♀️",
|
||||
"🕵🏻♀️", "🕵🏼♀️", "🕵🏽♀️", "🕵🏾♀️", "🕵🏿♀️", "💂", "💂🏻", "💂🏼", "💂🏽", "💂🏾", "💂🏿", "💂♂️",
|
||||
"💂🏻♂️", "💂🏼♂️", "💂🏽♂️", "💂🏾♂️", "💂🏿♂️", "💂♀️", "💂🏻♀️", "💂🏼♀️", "💂🏽♀️", "💂🏾♀️", "💂🏿♀️", "🥷",
|
||||
"🥷🏻", "🥷🏼", "🥷🏽", "🥷🏾", "🥷🏿", "👷", "👷🏻", "👷🏼", "👷🏽", "👷🏾", "👷🏿", "👷♂️",
|
||||
"👷🏻♂️", "👷🏼♂️", "👷🏽♂️", "👷🏾♂️", "👷🏿♂️", "👷♀️", "👷🏻♀️", "👷🏼♀️", "👷🏽♀️", "👷🏾♀️", "👷🏿♀️", "🫅",
|
||||
"🫅🏻", "🫅🏼", "🫅🏽", "🫅🏾", "🫅🏿", "🤴", "🤴🏻", "🤴🏼", "🤴🏽", "🤴🏾", "🤴🏿", "👸",
|
||||
"👸🏻", "👸🏼", "👸🏽", "👸🏾", "👸🏿", "👳", "👳🏻", "👳🏼", "👳🏽", "👳🏾", "👳🏿", "👳♂️",
|
||||
"👳🏻♂️", "👳🏼♂️", "👳🏽♂️", "👳🏾♂️", "👳🏿♂️", "👳♀️", "👳🏻♀️", "👳🏼♀️", "👳🏽♀️", "👳🏾♀️", "👳🏿♀️", "👲",
|
||||
"👲🏻", "👲🏼", "👲🏽", "👲🏾", "👲🏿", "🧕", "🧕🏻", "🧕🏼", "🧕🏽", "🧕🏾", "🧕🏿", "🤵",
|
||||
"🤵🏻", "🤵🏼", "🤵🏽", "🤵🏾", "🤵🏿", "🤵♂️", "🤵🏻♂️", "🤵🏼♂️", "🤵🏽♂️", "🤵🏾♂️", "🤵🏿♂️", "🤵♀️",
|
||||
"🤵🏻♀️", "🤵🏼♀️", "🤵🏽♀️", "🤵🏾♀️", "🤵🏿♀️", "👰", "👰🏻", "👰🏼", "👰🏽", "👰🏾", "👰🏿", "👰♂️",
|
||||
"👰🏻♂️", "👰🏼♂️", "👰🏽♂️", "👰🏾♂️", "👰🏿♂️", "👰♀️", "👰🏻♀️", "👰🏼♀️", "👰🏽♀️", "👰🏾♀️", "👰🏿♀️", "🤰",
|
||||
"🤰🏻", "🤰🏼", "🤰🏽", "🤰🏾", "🤰🏿", "🫃", "🫃🏻", "🫃🏼", "🫃🏽", "🫃🏾", "🫃🏿", "🫄",
|
||||
"🫄🏻", "🫄🏼", "🫄🏽", "🫄🏾", "🫄🏿", "🤱", "🤱🏻", "🤱🏼", "🤱🏽", "🤱🏾", "🤱🏿", "👩🍼",
|
||||
"👩🏻🍼", "👩🏼🍼", "👩🏽🍼", "👩🏾🍼", "👩🏿🍼", "👨🍼", "👨🏻🍼", "👨🏼🍼", "👨🏽🍼", "👨🏾🍼", "👨🏿🍼", "🧑🍼",
|
||||
"🧑🏻🍼", "🧑🏼🍼", "🧑🏽🍼", "🧑🏾🍼", "🧑🏿🍼", "👼", "👼🏻", "👼🏼", "👼🏽", "👼🏾", "👼🏿", "🎅",
|
||||
"🎅🏻", "🎅🏼", "🎅🏽", "🎅🏾", "🎅🏿", "🤶", "🤶🏻", "🤶🏼", "🤶🏽", "🤶🏾", "🤶🏿", "🧑🎄",
|
||||
"🧑🏻🎄", "🧑🏼🎄", "🧑🏽🎄", "🧑🏾🎄", "🧑🏿🎄", "🦸", "🦸🏻", "🦸🏼", "🦸🏽", "🦸🏾", "🦸🏿", "🦸♂️",
|
||||
"🦸🏻♂️", "🦸🏼♂️", "🦸🏽♂️", "🦸🏾♂️", "🦸🏿♂️", "🦸♀️", "🦸🏻♀️", "🦸🏼♀️", "🦸🏽♀️", "🦸🏾♀️", "🦸🏿♀️", "🦹",
|
||||
"🦹🏻", "🦹🏼", "🦹🏽", "🦹🏾", "🦹🏿", "🦹♂️", "🦹🏻♂️", "🦹🏼♂️", "🦹🏽♂️", "🦹🏾♂️", "🦹🏿♂️", "🦹♀️",
|
||||
"🦹🏻♀️", "🦹🏼♀️", "🦹🏽♀️", "🦹🏾♀️", "🦹🏿♀️", "🧙", "🧙🏻", "🧙🏼", "🧙🏽", "🧙🏾", "🧙🏿", "🧙♂️",
|
||||
"🧙🏻♂️", "🧙🏼♂️", "🧙🏽♂️", "🧙🏾♂️", "🧙🏿♂️", "🧙♀️", "🧙🏻♀️", "🧙🏼♀️", "🧙🏽♀️", "🧙🏾♀️", "🧙🏿♀️", "🧚",
|
||||
"🧚🏻", "🧚🏼", "🧚🏽", "🧚🏾", "🧚🏿", "🧚♂️", "🧚🏻♂️", "🧚🏼♂️", "🧚🏽♂️", "🧚🏾♂️", "🧚🏿♂️", "🧚♀️",
|
||||
"🧚🏻♀️", "🧚🏼♀️", "🧚🏽♀️", "🧚🏾♀️", "🧚🏿♀️", "🧛", "🧛🏻", "🧛🏼", "🧛🏽", "🧛🏾", "🧛🏿", "🧛♂️",
|
||||
"🧛🏻♂️", "🧛🏼♂️", "🧛🏽♂️", "🧛🏾♂️", "🧛🏿♂️", "🧛♀️", "🧛🏻♀️", "🧛🏼♀️", "🧛🏽♀️", "🧛🏾♀️", "🧛🏿♀️", "🧜",
|
||||
"🧜🏻", "🧜🏼", "🧜🏽", "🧜🏾", "🧜🏿", "🧜♂️", "🧜🏻♂️", "🧜🏼♂️", "🧜🏽♂️", "🧜🏾♂️", "🧜🏿♂️", "🧜♀️",
|
||||
"🧜🏻♀️", "🧜🏼♀️", "🧜🏽♀️", "🧜🏾♀️", "🧜🏿♀️", "🧝", "🧝🏻", "🧝🏼", "🧝🏽", "🧝🏾", "🧝🏿", "🧝♂️",
|
||||
"🧝🏻♂️", "🧝🏼♂️", "🧝🏽♂️", "🧝🏾♂️", "🧝🏿♂️", "🧝♀️", "🧝🏻♀️", "🧝🏼♀️", "🧝🏽♀️", "🧝🏾♀️", "🧝🏿♀️", "🧞",
|
||||
"🧞♂️", "🧞♀️", "🧟", "🧟♂️", "🧟♀️", "🧌", "", "💆", "💆🏻", "💆🏼", "💆🏽", "💆🏾",
|
||||
"💆🏿", "💆♂️", "💆🏻♂️", "💆🏼♂️", "💆🏽♂️", "💆🏾♂️", "💆🏿♂️", "💆♀️", "💆🏻♀️", "💆🏼♀️", "💆🏽♀️", "💆🏾♀️",
|
||||
"💆🏿♀️", "💇", "💇🏻", "💇🏼", "💇🏽", "💇🏾", "💇🏿", "💇♂️", "💇🏻♂️", "💇🏼♂️", "💇🏽♂️", "💇🏾♂️",
|
||||
"💇🏿♂️", "💇♀️", "💇🏻♀️", "💇🏼♀️", "💇🏽♀️", "💇🏾♀️", "💇🏿♀️", "🚶", "🚶🏻", "🚶🏼", "🚶🏽", "🚶🏾",
|
||||
"🚶🏿", "🚶♂️", "🚶🏻♂️", "🚶🏼♂️", "🚶🏽♂️", "🚶🏾♂️", "🚶🏿♂️", "🚶♀️", "🚶🏻♀️", "🚶🏼♀️", "🚶🏽♀️", "🚶🏾♀️",
|
||||
"🚶🏿♀️", "🚶➡️", "🚶🏻➡️", "🚶🏼➡️", "🚶🏽➡️", "🚶🏾➡️", "🚶🏿➡️", "🚶♀️➡️", "🚶🏻♀️➡️", "🚶🏼♀️➡️", "🚶🏽♀️➡️", "🚶🏾♀️➡️",
|
||||
"🚶🏿♀️➡️", "🚶♂️➡️", "🚶🏻♂️➡️", "🚶🏼♂️➡️", "🚶🏽♂️➡️", "🚶🏾♂️➡️", "🚶🏿♂️➡️", "🧍", "🧍🏻", "🧍🏼", "🧍🏽", "🧍🏾",
|
||||
"🧍🏿", "🧍♂️", "🧍🏻♂️", "🧍🏼♂️", "🧍🏽♂️", "🧍🏾♂️", "🧍🏿♂️", "🧍♀️", "🧍🏻♀️", "🧍🏼♀️", "🧍🏽♀️", "🧍🏾♀️",
|
||||
"🧍🏿♀️", "🧎", "🧎🏻", "🧎🏼", "🧎🏽", "🧎🏾", "🧎🏿", "🧎♂️", "🧎🏻♂️", "🧎🏼♂️", "🧎🏽♂️", "🧎🏾♂️",
|
||||
"🧎🏿♂️", "🧎♀️", "🧎🏻♀️", "🧎🏼♀️", "🧎🏽♀️", "🧎🏾♀️", "🧎🏿♀️", "🧎➡️", "🧎🏻➡️", "🧎🏼➡️", "🧎🏽➡️", "🧎🏾➡️",
|
||||
"🧎🏿➡️", "🧎♀️➡️", "🧎🏻♀️➡️", "🧎🏼♀️➡️", "🧎🏽♀️➡️", "🧎🏾♀️➡️", "🧎🏿♀️➡️", "🧎♂️➡️", "🧎🏻♂️➡️", "🧎🏼♂️➡️", "🧎🏽♂️➡️", "🧎🏾♂️➡️",
|
||||
"🧎🏿♂️➡️", "🧑🦯", "🧑🏻🦯", "🧑🏼🦯", "🧑🏽🦯", "🧑🏾🦯", "🧑🏿🦯", "🧑🦯➡️", "🧑🏻🦯➡️", "🧑🏼🦯➡️", "🧑🏽🦯➡️", "🧑🏾🦯➡️",
|
||||
"🧑🏿🦯➡️", "👨🦯", "👨🏻🦯", "👨🏼🦯", "👨🏽🦯", "👨🏾🦯", "👨🏿🦯", "👨🦯➡️", "👨🏻🦯➡️", "👨🏼🦯➡️", "👨🏽🦯➡️", "👨🏾🦯➡️",
|
||||
"👨🏿🦯➡️", "👩🦯", "👩🏻🦯", "👩🏼🦯", "👩🏽🦯", "👩🏾🦯", "👩🏿🦯", "👩🦯➡️", "👩🏻🦯➡️", "👩🏼🦯➡️", "👩🏽🦯➡️", "👩🏾🦯➡️",
|
||||
"👩🏿🦯➡️", "🧑🦼", "🧑🏻🦼", "🧑🏼🦼", "🧑🏽🦼", "🧑🏾🦼", "🧑🏿🦼", "🧑🦼➡️", "🧑🏻🦼➡️", "🧑🏼🦼➡️", "🧑🏽🦼➡️", "🧑🏾🦼➡️",
|
||||
"🧑🏿🦼➡️", "👨🦼", "👨🏻🦼", "👨🏼🦼", "👨🏽🦼", "👨🏾🦼", "👨🏿🦼", "👨🦼➡️", "👨🏻🦼➡️", "👨🏼🦼➡️", "👨🏽🦼➡️", "👨🏾🦼➡️",
|
||||
"👨🏿🦼➡️", "👩🦼", "👩🏻🦼", "👩🏼🦼", "👩🏽🦼", "👩🏾🦼", "👩🏿🦼", "👩🦼➡️", "👩🏻🦼➡️", "👩🏼🦼➡️", "👩🏽🦼➡️", "👩🏾🦼➡️",
|
||||
"👩🏿🦼➡️", "🧑🦽", "🧑🏻🦽", "🧑🏼🦽", "🧑🏽🦽", "🧑🏾🦽", "🧑🏿🦽", "🧑🦽➡️", "🧑🏻🦽➡️", "🧑🏼🦽➡️", "🧑🏽🦽➡️", "🧑🏾🦽➡️",
|
||||
"🧑🏿🦽➡️", "👨🦽", "👨🏻🦽", "👨🏼🦽", "👨🏽🦽", "👨🏾🦽", "👨🏿🦽", "👨🦽➡️", "👨🏻🦽➡️", "👨🏼🦽➡️", "👨🏽🦽➡️", "👨🏾🦽➡️",
|
||||
"👨🏿🦽➡️", "👩🦽", "👩🏻🦽", "👩🏼🦽", "👩🏽🦽", "👩🏾🦽", "👩🏿🦽", "👩🦽➡️", "👩🏻🦽➡️", "👩🏼🦽➡️", "👩🏽🦽➡️", "👩🏾🦽➡️",
|
||||
"👩🏿🦽➡️", "🏃", "🏃🏻", "🏃🏼", "🏃🏽", "🏃🏾", "🏃🏿", "🏃♂️", "🏃🏻♂️", "🏃🏼♂️", "🏃🏽♂️", "🏃🏾♂️",
|
||||
"🏃🏿♂️", "🏃♀️", "🏃🏻♀️", "🏃🏼♀️", "🏃🏽♀️", "🏃🏾♀️", "🏃🏿♀️", "🏃➡️", "🏃🏻➡️", "🏃🏼➡️", "🏃🏽➡️", "🏃🏾➡️",
|
||||
"🏃🏿➡️", "🏃♀️➡️", "🏃🏻♀️➡️", "🏃🏼♀️➡️", "🏃🏽♀️➡️", "🏃🏾♀️➡️", "🏃🏿♀️➡️", "🏃♂️➡️", "🏃🏻♂️➡️", "🏃🏼♂️➡️", "🏃🏽♂️➡️", "🏃🏾♂️➡️",
|
||||
"🏃🏿♂️➡️", "🧑🩰", "🧑🏻🩰", "🧑🏼🩰", "🧑🏽🩰", "🧑🏾🩰", "🧑🏿🩰", "💃", "💃🏻", "💃🏼", "💃🏽", "💃🏾",
|
||||
"💃🏿", "🕺", "🕺🏻", "🕺🏼", "🕺🏽", "🕺🏾", "🕺🏿", "🕴️", "🕴🏻", "🕴🏼", "🕴🏽", "🕴🏾",
|
||||
"🕴🏿", "👯", "👯🏻", "👯🏼", "👯🏽", "👯🏾", "👯🏿", "👯♂️", "👯🏻♂️", "👯🏼♂️", "👯🏽♂️", "👯🏾♂️",
|
||||
"👯🏿♂️", "👯♀️", "👯🏻♀️", "👯🏼♀️", "👯🏽♀️", "👯🏾♀️", "👯🏿♀️", "🧑🏻🐰🧑🏼", "🧑🏻🐰🧑🏽", "🧑🏻🐰🧑🏾", "🧑🏻🐰🧑🏿", "🧑🏼🐰🧑🏻",
|
||||
"🧑🏼🐰🧑🏽", "🧑🏼🐰🧑🏾", "🧑🏼🐰🧑🏿", "🧑🏽🐰🧑🏻", "🧑🏽🐰🧑🏼", "🧑🏽🐰🧑🏾", "🧑🏽🐰🧑🏿", "🧑🏾🐰🧑🏻", "🧑🏾🐰🧑🏼", "🧑🏾🐰🧑🏽", "🧑🏾🐰🧑🏿", "🧑🏿🐰🧑🏻",
|
||||
"🧑🏿🐰🧑🏼", "🧑🏿🐰🧑🏽", "🧑🏿🐰🧑🏾", "👨🏻🐰👨🏼", "👨🏻🐰👨🏽", "👨🏻🐰👨🏾", "👨🏻🐰👨🏿", "👨🏼🐰👨🏻", "👨🏼🐰👨🏽", "👨🏼🐰👨🏾", "👨🏼🐰👨🏿", "👨🏽🐰👨🏻",
|
||||
"👨🏽🐰👨🏼", "👨🏽🐰👨🏾", "👨🏽🐰👨🏿", "👨🏾🐰👨🏻", "👨🏾🐰👨🏼", "👨🏾🐰👨🏽", "👨🏾🐰👨🏿", "👨🏿🐰👨🏻", "👨🏿🐰👨🏼", "👨🏿🐰👨🏽", "👨🏿🐰👨🏾", "👩🏻🐰👩🏼",
|
||||
"👩🏻🐰👩🏽", "👩🏻🐰👩🏾", "👩🏻🐰👩🏿", "👩🏼🐰👩🏻", "👩🏼🐰👩🏽", "👩🏼🐰👩🏾", "👩🏼🐰👩🏿", "👩🏽🐰👩🏻", "👩🏽🐰👩🏼", "👩🏽🐰👩🏾", "👩🏽🐰👩🏿", "👩🏾🐰👩🏻",
|
||||
"👩🏾🐰👩🏼", "👩🏾🐰👩🏽", "👩🏾🐰👩🏿", "👩🏿🐰👩🏻", "👩🏿🐰👩🏼", "👩🏿🐰👩🏽", "👩🏿🐰👩🏾", "🧖", "🧖🏻", "🧖🏼", "🧖🏽", "🧖🏾",
|
||||
"🧖🏿", "🧖♂️", "🧖🏻♂️", "🧖🏼♂️", "🧖🏽♂️", "🧖🏾♂️", "🧖🏿♂️", "🧖♀️", "🧖🏻♀️", "🧖🏼♀️", "🧖🏽♀️", "🧖🏾♀️",
|
||||
"🧖🏿♀️", "🧗", "🧗🏻", "🧗🏼", "🧗🏽", "🧗🏾", "🧗🏿", "🧗♂️", "🧗🏻♂️", "🧗🏼♂️", "🧗🏽♂️", "🧗🏾♂️",
|
||||
"🧗🏿♂️", "🧗♀️", "🧗🏻♀️", "🧗🏼♀️", "🧗🏽♀️", "🧗🏾♀️", "🧗🏿♀️", "🤺", "🏇", "🏇🏻", "🏇🏼", "🏇🏽",
|
||||
"🏇🏾", "🏇🏿", "⛷️", "🏂", "🏂🏻", "🏂🏼", "🏂🏽", "🏂🏾", "🏂🏿", "🏌️", "🏌🏻", "🏌🏼",
|
||||
"🏌🏽", "🏌🏾", "🏌🏿", "🏌️♂️", "🏌🏻♂️", "🏌🏼♂️", "🏌🏽♂️", "🏌🏾♂️", "🏌🏿♂️", "🏌️♀️", "🏌🏻♀️", "🏌🏼♀️",
|
||||
"🏌🏽♀️", "🏌🏾♀️", "🏌🏿♀️", "🏄", "🏄🏻", "🏄🏼", "🏄🏽", "🏄🏾", "🏄🏿", "🏄♂️", "🏄🏻♂️", "🏄🏼♂️",
|
||||
"🏄🏽♂️", "🏄🏾♂️", "🏄🏿♂️", "🏄♀️", "🏄🏻♀️", "🏄🏼♀️", "🏄🏽♀️", "🏄🏾♀️", "🏄🏿♀️", "🚣", "🚣🏻", "🚣🏼",
|
||||
"🚣🏽", "🚣🏾", "🚣🏿", "🚣♂️", "🚣🏻♂️", "🚣🏼♂️", "🚣🏽♂️", "🚣🏾♂️", "🚣🏿♂️", "🚣♀️", "🚣🏻♀️", "🚣🏼♀️",
|
||||
"🚣🏽♀️", "🚣🏾♀️", "🚣🏿♀️", "🏊", "🏊🏻", "🏊🏼", "🏊🏽", "🏊🏾", "🏊🏿", "🏊♂️", "🏊🏻♂️", "🏊🏼♂️",
|
||||
"🏊🏽♂️", "🏊🏾♂️", "🏊🏿♂️", "🏊♀️", "🏊🏻♀️", "🏊🏼♀️", "🏊🏽♀️", "🏊🏾♀️", "🏊🏿♀️", "⛹️", "⛹🏻", "⛹🏼",
|
||||
"⛹🏽", "⛹🏾", "⛹🏿", "⛹️♂️", "⛹🏻♂️", "⛹🏼♂️", "⛹🏽♂️", "⛹🏾♂️", "⛹🏿♂️", "⛹️♀️", "⛹🏻♀️", "⛹🏼♀️",
|
||||
"⛹🏽♀️", "⛹🏾♀️", "⛹🏿♀️", "🏋️", "🏋🏻", "🏋🏼", "🏋🏽", "🏋🏾", "🏋🏿", "🏋️♂️", "🏋🏻♂️", "🏋🏼♂️",
|
||||
"🏋🏽♂️", "🏋🏾♂️", "🏋🏿♂️", "🏋️♀️", "🏋🏻♀️", "🏋🏼♀️", "🏋🏽♀️", "🏋🏾♀️", "🏋🏿♀️", "🚴", "🚴🏻", "🚴🏼",
|
||||
"🚴🏽", "🚴🏾", "🚴🏿", "🚴♂️", "🚴🏻♂️", "🚴🏼♂️", "🚴🏽♂️", "🚴🏾♂️", "🚴🏿♂️", "🚴♀️", "🚴🏻♀️", "🚴🏼♀️",
|
||||
"🚴🏽♀️", "🚴🏾♀️", "🚴🏿♀️", "🚵", "🚵🏻", "🚵🏼", "🚵🏽", "🚵🏾", "🚵🏿", "🚵♂️", "🚵🏻♂️", "🚵🏼♂️",
|
||||
"🚵🏽♂️", "🚵🏾♂️", "🚵🏿♂️", "🚵♀️", "🚵🏻♀️", "🚵🏼♀️", "🚵🏽♀️", "🚵🏾♀️", "🚵🏿♀️", "🤸", "🤸🏻", "🤸🏼",
|
||||
"🤸🏽", "🤸🏾", "🤸🏿", "🤸♂️", "🤸🏻♂️", "🤸🏼♂️", "🤸🏽♂️", "🤸🏾♂️", "🤸🏿♂️", "🤸♀️", "🤸🏻♀️", "🤸🏼♀️",
|
||||
"🤸🏽♀️", "🤸🏾♀️", "🤸🏿♀️", "🤼", "🤼🏻", "🤼🏼", "🤼🏽", "🤼🏾", "🤼🏿", "🤼♂️", "🤼🏻♂️", "🤼🏼♂️",
|
||||
"🤼🏽♂️", "🤼🏾♂️", "🤼🏿♂️", "🤼♀️", "🤼🏻♀️", "🤼🏼♀️", "🤼🏽♀️", "🤼🏾♀️", "🤼🏿♀️", "🧑🏻🧑🏼", "🧑🏻🧑🏽", "🧑🏻🧑🏾",
|
||||
"🧑🏻🧑🏿", "🧑🏼🧑🏻", "🧑🏼🧑🏽", "🧑🏼🧑🏾", "🧑🏼🧑🏿", "🧑🏽🧑🏻", "🧑🏽🧑🏼", "🧑🏽🧑🏾", "🧑🏽🧑🏿", "🧑🏾🧑🏻", "🧑🏾🧑🏼", "🧑🏾🧑🏽",
|
||||
"🧑🏾🧑🏿", "🧑🏿🧑🏻", "🧑🏿🧑🏼", "🧑🏿🧑🏽", "🧑🏿🧑🏾", "👨🏻👨🏼", "👨🏻👨🏽", "👨🏻👨🏾", "👨🏻👨🏿", "👨🏼👨🏻", "👨🏼👨🏽", "👨🏼👨🏾",
|
||||
"👨🏼👨🏿", "👨🏽👨🏻", "👨🏽👨🏼", "👨🏽👨🏾", "👨🏽👨🏿", "👨🏾👨🏻", "👨🏾👨🏼", "👨🏾👨🏽", "👨🏾👨🏿", "👨🏿👨🏻", "👨🏿👨🏼", "👨🏿👨🏽",
|
||||
"👨🏿👨🏾", "👩🏻👩🏼", "👩🏻👩🏽", "👩🏻👩🏾", "👩🏻👩🏿", "👩🏼👩🏻", "👩🏼👩🏽", "👩🏼👩🏾", "👩🏼👩🏿", "👩🏽👩🏻", "👩🏽👩🏼", "👩🏽👩🏾",
|
||||
"👩🏽👩🏿", "👩🏾👩🏻", "👩🏾👩🏼", "👩🏾👩🏽", "👩🏾👩🏿", "👩🏿👩🏻", "👩🏿👩🏼", "👩🏿👩🏽", "👩🏿👩🏾", "🤽", "🤽🏻", "🤽🏼",
|
||||
"🤽🏽", "🤽🏾", "🤽🏿", "🤽♂️", "🤽🏻♂️", "🤽🏼♂️", "🤽🏽♂️", "🤽🏾♂️", "🤽🏿♂️", "🤽♀️", "🤽🏻♀️", "🤽🏼♀️",
|
||||
"🤽🏽♀️", "🤽🏾♀️", "🤽🏿♀️", "🤾", "🤾🏻", "🤾🏼", "🤾🏽", "🤾🏾", "🤾🏿", "🤾♂️", "🤾🏻♂️", "🤾🏼♂️",
|
||||
"🤾🏽♂️", "🤾🏾♂️", "🤾🏿♂️", "🤾♀️", "🤾🏻♀️", "🤾🏼♀️", "🤾🏽♀️", "🤾🏾♀️", "🤾🏿♀️", "🤹", "🤹🏻", "🤹🏼",
|
||||
"🤹🏽", "🤹🏾", "🤹🏿", "🤹♂️", "🤹🏻♂️", "🤹🏼♂️", "🤹🏽♂️", "🤹🏾♂️", "🤹🏿♂️", "🤹♀️", "🤹🏻♀️", "🤹🏼♀️",
|
||||
"🤹🏽♀️", "🤹🏾♀️", "🤹🏿♀️", "🧘", "🧘🏻", "🧘🏼", "🧘🏽", "🧘🏾", "🧘🏿", "🧘♂️", "🧘🏻♂️", "🧘🏼♂️",
|
||||
"🧘🏽♂️", "🧘🏾♂️", "🧘🏿♂️", "🧘♀️", "🧘🏻♀️", "🧘🏼♀️", "🧘🏽♀️", "🧘🏾♀️", "🧘🏿♀️", "🛀", "🛀🏻", "🛀🏼",
|
||||
"🛀🏽", "🛀🏾", "🛀🏿", "🛌", "🛌🏻", "🛌🏼", "🛌🏽", "🛌🏾", "🛌🏿", "🧑🤝🧑", "🧑🏻🤝🧑🏻", "🧑🏻🤝🧑🏼",
|
||||
"🧑🏻🤝🧑🏽", "🧑🏻🤝🧑🏾", "🧑🏻🤝🧑🏿", "🧑🏼🤝🧑🏻", "🧑🏼🤝🧑🏼", "🧑🏼🤝🧑🏽", "🧑🏼🤝🧑🏾", "🧑🏼🤝🧑🏿", "🧑🏽🤝🧑🏻", "🧑🏽🤝🧑🏼", "🧑🏽🤝🧑🏽", "🧑🏽🤝🧑🏾",
|
||||
"🧑🏽🤝🧑🏿", "🧑🏾🤝🧑🏻", "🧑🏾🤝🧑🏼", "🧑🏾🤝🧑🏽", "🧑🏾🤝🧑🏾", "🧑🏾🤝🧑🏿", "🧑🏿🤝🧑🏻", "🧑🏿🤝🧑🏼", "🧑🏿🤝🧑🏽", "🧑🏿🤝🧑🏾", "🧑🏿🤝🧑🏿", "👭",
|
||||
"👭🏻", "👩🏻🤝👩🏼", "👩🏻🤝👩🏽", "👩🏻🤝👩🏾", "👩🏻🤝👩🏿", "👩🏼🤝👩🏻", "👭🏼", "👩🏼🤝👩🏽", "👩🏼🤝👩🏾", "👩🏼🤝👩🏿", "👩🏽🤝👩🏻", "👩🏽🤝👩🏼",
|
||||
"👭🏽", "👩🏽🤝👩🏾", "👩🏽🤝👩🏿", "👩🏾🤝👩🏻", "👩🏾🤝👩🏼", "👩🏾🤝👩🏽", "👭🏾", "👩🏾🤝👩🏿", "👩🏿🤝👩🏻", "👩🏿🤝👩🏼", "👩🏿🤝👩🏽", "👩🏿🤝👩🏾",
|
||||
"👭🏿", "👫", "👫🏻", "👩🏻🤝👨🏼", "👩🏻🤝👨🏽", "👩🏻🤝👨🏾", "👩🏻🤝👨🏿", "👩🏼🤝👨🏻", "👫🏼", "👩🏼🤝👨🏽", "👩🏼🤝👨🏾", "👩🏼🤝👨🏿",
|
||||
"👩🏽🤝👨🏻", "👩🏽🤝👨🏼", "👫🏽", "👩🏽🤝👨🏾", "👩🏽🤝👨🏿", "👩🏾🤝👨🏻", "👩🏾🤝👨🏼", "👩🏾🤝👨🏽", "👫🏾", "👩🏾🤝👨🏿", "👩🏿🤝👨🏻", "👩🏿🤝👨🏼",
|
||||
"👩🏿🤝👨🏽", "👩🏿🤝👨🏾", "👫🏿", "👬", "👬🏻", "👨🏻🤝👨🏼", "👨🏻🤝👨🏽", "👨🏻🤝👨🏾", "👨🏻🤝👨🏿", "👨🏼🤝👨🏻", "👬🏼", "👨🏼🤝👨🏽",
|
||||
"👨🏼🤝👨🏾", "👨🏼🤝👨🏿", "👨🏽🤝👨🏻", "👨🏽🤝👨🏼", "👬🏽", "👨🏽🤝👨🏾", "👨🏽🤝👨🏿", "👨🏾🤝👨🏻", "👨🏾🤝👨🏼", "👨🏾🤝👨🏽", "👬🏾", "👨🏾🤝👨🏿",
|
||||
"👨🏿🤝👨🏻", "👨🏿🤝👨🏼", "👨🏿🤝👨🏽", "👨🏿🤝👨🏾", "👬🏿", "💏", "💏🏻", "💏🏼", "💏🏽", "💏🏾", "💏🏿", "🧑🏻❤️💋🧑🏼",
|
||||
"🧑🏻❤️💋🧑🏽", "🧑🏻❤️💋🧑🏾", "🧑🏻❤️💋🧑🏿", "🧑🏼❤️💋🧑🏻", "🧑🏼❤️💋🧑🏽", "🧑🏼❤️💋🧑🏾", "🧑🏼❤️💋🧑🏿", "🧑🏽❤️💋🧑🏻", "🧑🏽❤️💋🧑🏼", "🧑🏽❤️💋🧑🏾", "🧑🏽❤️💋🧑🏿", "🧑🏾❤️💋🧑🏻",
|
||||
"🧑🏾❤️💋🧑🏼", "🧑🏾❤️💋🧑🏽", "🧑🏾❤️💋🧑🏿", "🧑🏿❤️💋🧑🏻", "🧑🏿❤️💋🧑🏼", "🧑🏿❤️💋🧑🏽", "🧑🏿❤️💋🧑🏾", "👩❤️💋👨", "👩🏻❤️💋👨🏻", "👩🏻❤️💋👨🏼", "👩🏻❤️💋👨🏽", "👩🏻❤️💋👨🏾",
|
||||
"👩🏻❤️💋👨🏿", "👩🏼❤️💋👨🏻", "👩🏼❤️💋👨🏼", "👩🏼❤️💋👨🏽", "👩🏼❤️💋👨🏾", "👩🏼❤️💋👨🏿", "👩🏽❤️💋👨🏻", "👩🏽❤️💋👨🏼", "👩🏽❤️💋👨🏽", "👩🏽❤️💋👨🏾", "👩🏽❤️💋👨🏿", "👩🏾❤️💋👨🏻",
|
||||
"👩🏾❤️💋👨🏼", "👩🏾❤️💋👨🏽", "👩🏾❤️💋👨🏾", "👩🏾❤️💋👨🏿", "👩🏿❤️💋👨🏻", "👩🏿❤️💋👨🏼", "👩🏿❤️💋👨🏽", "👩🏿❤️💋👨🏾", "👩🏿❤️💋👨🏿", "👨❤️💋👨", "👨🏻❤️💋👨🏻", "👨🏻❤️💋👨🏼",
|
||||
"👨🏻❤️💋👨🏽", "👨🏻❤️💋👨🏾", "👨🏻❤️💋👨🏿", "👨🏼❤️💋👨🏻", "👨🏼❤️💋👨🏼", "👨🏼❤️💋👨🏽", "👨🏼❤️💋👨🏾", "👨🏼❤️💋👨🏿", "👨🏽❤️💋👨🏻", "👨🏽❤️💋👨🏼", "👨🏽❤️💋👨🏽", "👨🏽❤️💋👨🏾",
|
||||
"👨🏽❤️💋👨🏿", "👨🏾❤️💋👨🏻", "👨🏾❤️💋👨🏼", "👨🏾❤️💋👨🏽", "👨🏾❤️💋👨🏾", "👨🏾❤️💋👨🏿", "👨🏿❤️💋👨🏻", "👨🏿❤️💋👨🏼", "👨🏿❤️💋👨🏽", "👨🏿❤️💋👨🏾", "👨🏿❤️💋👨🏿", "👩❤️💋👩",
|
||||
"👩🏻❤️💋👩🏻", "👩🏻❤️💋👩🏼", "👩🏻❤️💋👩🏽", "👩🏻❤️💋👩🏾", "👩🏻❤️💋👩🏿", "👩🏼❤️💋👩🏻", "👩🏼❤️💋👩🏼", "👩🏼❤️💋👩🏽", "👩🏼❤️💋👩🏾", "👩🏼❤️💋👩🏿", "👩🏽❤️💋👩🏻", "👩🏽❤️💋👩🏼",
|
||||
"👩🏽❤️💋👩🏽", "👩🏽❤️💋👩🏾", "👩🏽❤️💋👩🏿", "👩🏾❤️💋👩🏻", "👩🏾❤️💋👩🏼", "👩🏾❤️💋👩🏽", "👩🏾❤️💋👩🏾", "👩🏾❤️💋👩🏿", "👩🏿❤️💋👩🏻", "👩🏿❤️💋👩🏼", "👩🏿❤️💋👩🏽", "👩🏿❤️💋👩🏾",
|
||||
"👩🏿❤️💋👩🏿", "💑", "💑🏻", "💑🏼", "💑🏽", "💑🏾", "💑🏿", "🧑🏻❤️🧑🏼", "🧑🏻❤️🧑🏽", "🧑🏻❤️🧑🏾", "🧑🏻❤️🧑🏿", "🧑🏼❤️🧑🏻",
|
||||
"🧑🏼❤️🧑🏽", "🧑🏼❤️🧑🏾", "🧑🏼❤️🧑🏿", "🧑🏽❤️🧑🏻", "🧑🏽❤️🧑🏼", "🧑🏽❤️🧑🏾", "🧑🏽❤️🧑🏿", "🧑🏾❤️🧑🏻", "🧑🏾❤️🧑🏼", "🧑🏾❤️🧑🏽", "🧑🏾❤️🧑🏿", "🧑🏿❤️🧑🏻",
|
||||
"🧑🏿❤️🧑🏼", "🧑🏿❤️🧑🏽", "🧑🏿❤️🧑🏾", "👩❤️👨", "👩🏻❤️👨🏻", "👩🏻❤️👨🏼", "👩🏻❤️👨🏽", "👩🏻❤️👨🏾", "👩🏻❤️👨🏿", "👩🏼❤️👨🏻", "👩🏼❤️👨🏼", "👩🏼❤️👨🏽",
|
||||
"👩🏼❤️👨🏾", "👩🏼❤️👨🏿", "👩🏽❤️👨🏻", "👩🏽❤️👨🏼", "👩🏽❤️👨🏽", "👩🏽❤️👨🏾", "👩🏽❤️👨🏿", "👩🏾❤️👨🏻", "👩🏾❤️👨🏼", "👩🏾❤️👨🏽", "👩🏾❤️👨🏾", "👩🏾❤️👨🏿",
|
||||
"👩🏿❤️👨🏻", "👩🏿❤️👨🏼", "👩🏿❤️👨🏽", "👩🏿❤️👨🏾", "👩🏿❤️👨🏿", "👨❤️👨", "👨🏻❤️👨🏻", "👨🏻❤️👨🏼", "👨🏻❤️👨🏽", "👨🏻❤️👨🏾", "👨🏻❤️👨🏿", "👨🏼❤️👨🏻",
|
||||
"👨🏼❤️👨🏼", "👨🏼❤️👨🏽", "👨🏼❤️👨🏾", "👨🏼❤️👨🏿", "👨🏽❤️👨🏻", "👨🏽❤️👨🏼", "👨🏽❤️👨🏽", "👨🏽❤️👨🏾", "👨🏽❤️👨🏿", "👨🏾❤️👨🏻", "👨🏾❤️👨🏼", "👨🏾❤️👨🏽",
|
||||
"👨🏾❤️👨🏾", "👨🏾❤️👨🏿", "👨🏿❤️👨🏻", "👨🏿❤️👨🏼", "👨🏿❤️👨🏽", "👨🏿❤️👨🏾", "👨🏿❤️👨🏿", "👩❤️👩", "👩🏻❤️👩🏻", "👩🏻❤️👩🏼", "👩🏻❤️👩🏽", "👩🏻❤️👩🏾",
|
||||
"👩🏻❤️👩🏿", "👩🏼❤️👩🏻", "👩🏼❤️👩🏼", "👩🏼❤️👩🏽", "👩🏼❤️👩🏾", "👩🏼❤️👩🏿", "👩🏽❤️👩🏻", "👩🏽❤️👩🏼", "👩🏽❤️👩🏽", "👩🏽❤️👩🏾", "👩🏽❤️👩🏿", "👩🏾❤️👩🏻",
|
||||
"👩🏾❤️👩🏼", "👩🏾❤️👩🏽", "👩🏾❤️👩🏾", "👩🏾❤️👩🏿", "👩🏿❤️👩🏻", "👩🏿❤️👩🏼", "👩🏿❤️👩🏽", "👩🏿❤️👩🏾", "👩🏿❤️👩🏿", "👨👩👦", "👨👩👧", "👨👩👧👦",
|
||||
"👨👩👦👦", "👨👩👧👧", "👨👨👦", "👨👨👧", "👨👨👧👦", "👨👨👦👦", "👨👨👧👧", "👩👩👦", "👩👩👧", "👩👩👧👦", "👩👩👦👦", "👩👩👧👧",
|
||||
"👨👦", "👨👦👦", "👨👧", "👨👧👦", "👨👧👧", "👩👦", "👩👦👦", "👩👧", "👩👧👦", "👩👧👧", "🗣️", "👤",
|
||||
"👥", "🫂", "👪", "🧑🧑🧒", "🧑🧑🧒🧒", "🧑🧒", "🧑🧒🧒", "👣", "", "🐵", "🐒", "🦍",
|
||||
"🦧", "🐶", "🐕", "🦮", "🐕🦺", "🐩", "🐺", "🦊", "🦝", "🐱", "🐈", "🐈⬛",
|
||||
"🦁", "🐯", "🐅", "🐆", "🐴", "🫎", "🫏", "🐎", "🦄", "🦓", "🦌", "🦬",
|
||||
"🐮", "🐂", "🐃", "🐄", "🐷", "🐖", "🐗", "🐽", "🐏", "🐑", "🐐", "🐪",
|
||||
"🐫", "🦙", "🦒", "🐘", "🦣", "🦏", "🦛", "🐭", "🐁", "🐀", "🐹", "🐰",
|
||||
"🐇", "🐿️", "🦫", "🦔", "🦇", "🐻", "🐻❄️", "🐨", "🐼", "🦥", "🦦", "🦨",
|
||||
"🦘", "🦡", "🐾", "🦃", "🐔", "🐓", "🐣", "🐤", "🐥", "🐦", "🐧", "🕊️",
|
||||
"🦅", "🦆", "🦢", "🦉", "🦤", "🪶", "🦩", "🦚", "🦜", "🪽", "🐦⬛", "🪿",
|
||||
"🐦🔥", "🐸", "🐊", "🐢", "🦎", "🐍", "🐲", "🐉", "🦕", "🦖", "🐳", "🐋",
|
||||
"🐬", "", "🦭", "🐟", "🐠", "🐡", "🦈", "🐙", "🐚", "🪸", "🪼", "🦀",
|
||||
"🦞", "🦐", "🦑", "🦪", "🐌", "🦋", "🐛", "🐜", "🐝", "🪲", "🐞", "🦗",
|
||||
"🪳", "🕷️", "🕸️", "🦂", "🦟", "🪰", "🪱", "🦠", "💐", "🌸", "💮", "🪷",
|
||||
"🏵️", "🌹", "🥀", "🌺", "🌻", "🌼", "🌷", "🪻", "🌱", "🪴", "🌲", "🌳",
|
||||
"🌴", "🌵", "🌾", "🌿", "☘️", "🍀", "🍁", "🍂", "🍃", "🪹", "🪺", "🍄",
|
||||
"", "🍇", "🍈", "🍉", "🍊", "🍋", "🍋🟩", "🍌", "🍍", "🥭", "🍎", "🍏",
|
||||
"🍐", "🍑", "🍒", "🍓", "🫐", "🥝", "🍅", "🫒", "🥥", "🥑", "🍆", "🥔",
|
||||
"🥕", "🌽", "🌶️", "🫑", "🥒", "🥬", "🥦", "🧄", "🧅", "🥜", "🫘", "🌰",
|
||||
"🫚", "🫛", "🍄🟫", "", "🍞", "🥐", "🥖", "🫓", "🥨", "🥯", "🥞", "🧇",
|
||||
"🧀", "🍖", "🍗", "🥩", "🥓", "🍔", "🍟", "🍕", "🌭", "🥪", "🌮", "🌯",
|
||||
"🫔", "🥙", "🧆", "🥚", "🍳", "🥘", "🍲", "🫕", "🥣", "🥗", "🍿", "🧈",
|
||||
"🧂", "🥫", "🍱", "🍘", "🍙", "🍚", "🍛", "🍜", "🍝", "🍠", "🍢", "🍣",
|
||||
"🍤", "🍥", "🥮", "🍡", "🥟", "🥠", "🥡", "🍦", "🍧", "🍨", "🍩", "🍪",
|
||||
"🎂", "🍰", "🧁", "🥧", "🍫", "🍬", "🍭", "🍮", "🍯", "🍼", "🥛", "☕",
|
||||
"🫖", "🍵", "🍶", "🍾", "🍷", "🍸", "🍹", "🍺", "🍻", "🥂", "🥃", "🫗",
|
||||
"🥤", "🧋", "🧃", "🧉", "🧊", "🥢", "🍽️", "🍴", "🥄", "🔪", "🫙", "🏺",
|
||||
"🌍", "🌎", "🌏", "🌐", "🗺️", "🗾", "🧭", "🏔️", "⛰️", "", "🌋", "🗻",
|
||||
"🏕️", "🏖️", "🏜️", "🏝️", "🏞️", "🏟️", "🏛️", "🏗️", "🧱", "🪨", "🪵", "🛖",
|
||||
"🏘️", "🏚️", "🏠", "🏡", "🏢", "🏣", "🏤", "🏥", "🏦", "🏨", "🏩", "🏪",
|
||||
"🏫", "🏬", "🏭", "🏯", "🏰", "💒", "🗼", "🗽", "⛪", "🕌", "🛕", "🕍",
|
||||
"⛩️", "🕋", "⛲", "⛺", "🌁", "🌃", "🏙️", "🌄", "🌅", "🌆", "🌇", "🌉",
|
||||
"♨️", "🎠", "🛝", "🎡", "🎢", "💈", "🎪", "🚂", "🚃", "🚄", "🚅", "🚆",
|
||||
"🚇", "🚈", "🚉", "🚊", "🚝", "🚞", "🚋", "🚌", "🚍", "🚎", "🚐", "🚑",
|
||||
"🚒", "🚓", "🚔", "🚕", "🚖", "🚗", "🚘", "🚙", "🛻", "🚚", "🚛", "🚜",
|
||||
"🏎️", "🏍️", "🛵", "🦽", "🦼", "🛺", "🚲", "🛴", "🛹", "🛼", "🚏", "🛣️",
|
||||
"🛤️", "🛢️", "⛽", "🛞", "🚨", "🚥", "🚦", "🛑", "🚧", "⚓", "🛟", "⛵",
|
||||
"🛶", "🚤", "🛳️", "⛴️", "🛥️", "🚢", "✈️", "🛩️", "🛫", "🛬", "🪂", "💺",
|
||||
"🚁", "🚟", "🚠", "🚡", "🛰️", "🚀", "🛸", "🛎️", "🧳", "⌛", "⏳", "⌚",
|
||||
"⏰", "⏱️", "⏲️", "🕰️", "🕛", "🕧", "🕐", "🕜", "🕑", "🕝", "🕒", "🕞",
|
||||
"🕓", "🕟", "🕔", "🕠", "🕕", "🕡", "🕖", "🕢", "🕗", "🕣", "🕘", "🕤",
|
||||
"🕙", "🕥", "🕚", "🕦", "🌑", "🌒", "🌓", "🌔", "🌕", "🌖", "🌗", "🌘",
|
||||
"🌙", "🌚", "🌛", "🌜", "🌡️", "☀️", "🌝", "🌞", "🪐", "⭐", "🌟", "🌠",
|
||||
"🌌", "☁️", "⛅", "⛈️", "🌤️", "🌥️", "🌦️", "🌧️", "🌨️", "🌩️", "🌪️", "🌫️",
|
||||
"🌬️", "🌀", "🌈", "🌂", "☂️", "☔", "⛱️", "⚡", "❄️", "☃️", "⛄", "☄️",
|
||||
"🔥", "💧", "🌊", "🎃", "🎄", "🎆", "🎇", "🧨", "✨", "🎈", "🎉", "🎊",
|
||||
"🎋", "🎍", "🎎", "🎏", "🎐", "🎑", "🧧", "🎀", "🎁", "🎗️", "🎟️", "🎫",
|
||||
"🎖️", "🏆", "🏅", "🥇", "🥈", "🥉", "⚽", "⚾", "🥎", "🏀", "🏐", "🏈",
|
||||
"🏉", "🎾", "🥏", "🎳", "🏏", "🏑", "🏒", "🥍", "🏓", "🏸", "🥊", "🥋",
|
||||
"🥅", "⛳", "⛸️", "🎣", "🤿", "🎽", "🎿", "🛷", "🥌", "🎯", "🪀", "🪁",
|
||||
"🔫", "🎱", "🔮", "🪄", "🎮", "🕹️", "🎰", "🎲", "🧩", "🧸", "🪅", "🪩",
|
||||
"🪆", "♠️", "♥️", "♦️", "♣️", "♟️", "🃏", "🀄", "🎴", "🎭", "🖼️", "🎨",
|
||||
"🧵", "🪡", "🧶", "🪢", "👓", "🕶️", "🥽", "🥼", "🦺", "👔", "👕", "👖",
|
||||
"🧣", "🧤", "🧥", "🧦", "👗", "👘", "🥻", "🩱", "🩲", "🩳", "👙", "👚",
|
||||
"🪭", "👛", "👜", "👝", "🛍️", "🎒", "🩴", "👞", "👟", "🥾", "🥿", "👠",
|
||||
"👡", "🩰", "👢", "🪮", "👑", "👒", "🎩", "🎓", "🧢", "🪖", "⛑️", "📿",
|
||||
"💄", "💍", "💎", "🔇", "🔈", "🔉", "🔊", "📢", "📣", "📯", "🔔", "🔕",
|
||||
"🎼", "🎵", "🎶", "🎙️", "🎚️", "🎛️", "🎤", "🎧", "📻", "🎷", "🎺", "",
|
||||
"🪗", "🎸", "🎹", "🎻", "🪕", "🥁", "🪘", "🪇", "🪈", "", "📱", "📲",
|
||||
"☎️", "📞", "📟", "📠", "🔋", "🪫", "🔌", "💻", "🖥️", "🖨️", "⌨️", "🖱️",
|
||||
"🖲️", "💽", "💾", "💿", "📀", "🧮", "🎥", "🎞️", "📽️", "🎬", "📺", "📷",
|
||||
"📸", "📹", "📼", "🔍", "🔎", "🕯️", "💡", "🔦", "🏮", "🪔", "📔", "📕",
|
||||
"📖", "📗", "📘", "📙", "📚", "📓", "📒", "📃", "📜", "📄", "📰", "🗞️",
|
||||
"📑", "🔖", "🏷️", "🪙", "💰", "", "💴", "💵", "💶", "💷", "💸", "💳",
|
||||
"🧾", "💹", "✉️", "📧", "📨", "📩", "📤", "📥", "📦", "📫", "📪", "📬",
|
||||
"📭", "📮", "🗳️", "✏️", "✒️", "🖋️", "🖊️", "🖌️", "🖍️", "📝", "💼", "📁",
|
||||
"📂", "🗂️", "📅", "📆", "🗒️", "🗓️", "📇", "📈", "📉", "📊", "📋", "📌",
|
||||
"📍", "📎", "🖇️", "📏", "📐", "✂️", "🗃️", "🗄️", "🗑️", "🔒", "🔓", "🔏",
|
||||
"🔐", "🔑", "🗝️", "🔨", "🪓", "⛏️", "⚒️", "🛠️", "🗡️", "⚔️", "💣", "🪃",
|
||||
"🏹", "🛡️", "🪚", "🔧", "🪛", "🔩", "⚙️", "🗜️", "⚖️", "🦯", "🔗", "⛓️💥",
|
||||
"⛓️", "🪝", "🧰", "🧲", "🪜", "", "⚗️", "🧪", "🧫", "🧬", "🔬", "🔭",
|
||||
"📡", "💉", "🩸", "💊", "🩹", "🩼", "🩺", "🩻", "🚪", "🛗", "🪞", "🪟",
|
||||
"🛏️", "🛋️", "🪑", "🚽", "🪠", "🚿", "🛁", "🪤", "🪒", "🧴", "🧷", "🧹",
|
||||
"🧺", "🧻", "🪣", "🧼", "🫧", "🪥", "🧽", "🧯", "🛒", "🚬", "⚰️", "🪦",
|
||||
"⚱️", "🧿", "🪬", "🗿", "🪧", "🪪", "🏧", "🚮", "🚰", "♿", "🚹", "🚺",
|
||||
"🚻", "🚼", "🚾", "🛂", "🛃", "🛄", "🛅", "⚠️", "🚸", "⛔", "🚫", "🚳",
|
||||
"🚭", "🚯", "🚱", "🚷", "📵", "🔞", "☢️", "☣️", "⬆️", "↗️", "➡️", "↘️",
|
||||
"⬇️", "↙️", "⬅️", "↖️", "↕️", "↔️", "↩️", "↪️", "⤴️", "⤵️", "🔃", "🔄",
|
||||
"🔙", "🔚", "🔛", "🔜", "🔝", "🛐", "⚛️", "🕉️", "✡️", "☸️", "☯️", "✝️",
|
||||
"☦️", "☪️", "☮️", "🕎", "🔯", "🪯", "♈", "♉", "♊", "♋", "♌", "♍",
|
||||
"♎", "♏", "♐", "♑", "♒", "♓", "⛎", "🔀", "🔁", "🔂", "▶️", "⏩",
|
||||
"⏭️", "⏯️", "◀️", "⏪", "⏮️", "🔼", "⏫", "🔽", "⏬", "⏸️", "⏹️", "⏺️",
|
||||
"⏏️", "🎦", "🔅", "🔆", "📶", "🛜", "📳", "📴", "♀️", "♂️", "⚧️", "✖️",
|
||||
"➕", "➖", "➗", "🟰", "♾️", "‼️", "⁉️", "❓", "❔", "❕", "❗", "〰️",
|
||||
"💱", "💲", "⚕️", "♻️", "⚜️", "🔱", "📛", "🔰", "⭕", "✅", "☑️", "✔️",
|
||||
"❌", "❎", "➰", "➿", "〽️", "✳️", "✴️", "❇️", "©️", "®️", "™️", "",
|
||||
"#️⃣", "*️⃣", "0️⃣", "1️⃣", "2️⃣", "3️⃣", "4️⃣", "5️⃣", "6️⃣", "7️⃣", "8️⃣", "9️⃣",
|
||||
"🔟", "🔠", "🔡", "🔢", "🔣", "🔤", "🅰️", "🆎", "🅱️", "🆑", "🆒", "🆓",
|
||||
"ℹ️", "🆔", "Ⓜ️", "🆕", "🆖", "🅾️", "🆗", "🅿️", "🆘", "🆙", "🆚", "🈁",
|
||||
"🈂️", "🈷️", "🈶", "🈯", "🉐", "🈹", "🈚", "🈲", "🉑", "🈸", "🈴", "🈳",
|
||||
"㊗️", "㊙️", "🈺", "🈵", "🔴", "🟠", "🟡", "🟢", "🔵", "🟣", "🟤", "⚫",
|
||||
"⚪", "🟥", "🟧", "🟨", "🟩", "🟦", "🟪", "🟫", "⬛", "⬜", "◼️", "◻️",
|
||||
"◾", "◽", "▪️", "▫️", "🔶", "🔷", "🔸", "🔹", "🔺", "🔻", "💠", "🔘",
|
||||
"🔳", "🔲", "🏁", "🚩", "🎌", "🏴", "🏳️", "🏳️🌈", "🏳️⚧️", "🏴☠️", "🇦🇨", "🇦🇩",
|
||||
"🇦🇪", "🇦🇫", "🇦🇬", "🇦🇮", "🇦🇱", "🇦🇲", "🇦🇴", "🇦🇶", "🇦🇷", "🇦🇸", "🇦🇹", "🇦🇺",
|
||||
"🇦🇼", "🇦🇽", "🇦🇿", "🇧🇦", "🇧🇧", "🇧🇩", "🇧🇪", "🇧🇫", "🇧🇬", "🇧🇭", "🇧🇮", "🇧🇯",
|
||||
"🇧🇱", "🇧🇲", "🇧🇳", "🇧🇴", "🇧🇶", "🇧🇷", "🇧🇸", "🇧🇹", "🇧🇻", "🇧🇼", "🇧🇾", "🇧🇿",
|
||||
"🇨🇦", "🇨🇨", "🇨🇩", "🇨🇫", "🇨🇬", "🇨🇭", "🇨🇮", "🇨🇰", "🇨🇱", "🇨🇲", "🇨🇳", "🇨🇴",
|
||||
"🇨🇵", "🇨🇶", "🇨🇷", "🇨🇺", "🇨🇻", "🇨🇼", "🇨🇽", "🇨🇾", "🇨🇿", "🇩🇪", "🇩🇬", "🇩🇯",
|
||||
"🇩🇰", "🇩🇲", "🇩🇴", "🇩🇿", "🇪🇦", "🇪🇨", "🇪🇪", "🇪🇬", "🇪🇭", "🇪🇷", "🇪🇸", "🇪🇹",
|
||||
"🇪🇺", "🇫🇮", "🇫🇯", "🇫🇰", "🇫🇲", "🇫🇴", "🇫🇷", "🇬🇦", "🇬🇧", "🇬🇩", "🇬🇪", "🇬🇫",
|
||||
"🇬🇬", "🇬🇭", "🇬🇮", "🇬🇱", "🇬🇲", "🇬🇳", "🇬🇵", "🇬🇶", "🇬🇷", "🇬🇸", "🇬🇹", "🇬🇺",
|
||||
"🇬🇼", "🇬🇾", "🇭🇰", "🇭🇲", "🇭🇳", "🇭🇷", "🇭🇹", "🇭🇺", "🇮🇨", "🇮🇩", "🇮🇪", "🇮🇱",
|
||||
"🇮🇲", "🇮🇳", "🇮🇴", "🇮🇶", "🇮🇷", "🇮🇸", "🇮🇹", "🇯🇪", "🇯🇲", "🇯🇴", "🇯🇵", "🇰🇪",
|
||||
"🇰🇬", "🇰🇭", "🇰🇮", "🇰🇲", "🇰🇳", "🇰🇵", "🇰🇷", "🇰🇼", "🇰🇾", "🇰🇿", "🇱🇦", "🇱🇧",
|
||||
"🇱🇨", "🇱🇮", "🇱🇰", "🇱🇷", "🇱🇸", "🇱🇹", "🇱🇺", "🇱🇻", "🇱🇾", "🇲🇦", "🇲🇨", "🇲🇩",
|
||||
"🇲🇪", "🇲🇫", "🇲🇬", "🇲🇭", "🇲🇰", "🇲🇱", "🇲🇲", "🇲🇳", "🇲🇴", "🇲🇵", "🇲🇶", "🇲🇷",
|
||||
"🇲🇸", "🇲🇹", "🇲🇺", "🇲🇻", "🇲🇼", "🇲🇽", "🇲🇾", "🇲🇿", "🇳🇦", "🇳🇨", "🇳🇪", "🇳🇫",
|
||||
"🇳🇬", "🇳🇮", "🇳🇱", "🇳🇴", "🇳🇵", "🇳🇷", "🇳🇺", "🇳🇿", "🇴🇲", "🇵🇦", "🇵🇪", "🇵🇫",
|
||||
"🇵🇬", "🇵🇭", "🇵🇰", "🇵🇱", "🇵🇲", "🇵🇳", "🇵🇷", "🇵🇸", "🇵🇹", "🇵🇼", "🇵🇾", "🇶🇦",
|
||||
"🇷🇪", "🇷🇴", "🇷🇸", "🇷🇺", "🇷🇼", "🇸🇦", "🇸🇧", "🇸🇨", "🇸🇩", "🇸🇪", "🇸🇬", "🇸🇭",
|
||||
"🇸🇮", "🇸🇯", "🇸🇰", "🇸🇱", "🇸🇲", "🇸🇳", "🇸🇴", "🇸🇷", "🇸🇸", "🇸🇹", "🇸🇻", "🇸🇽",
|
||||
"🇸🇾", "🇸🇿", "🇹🇦", "🇹🇨", "🇹🇩", "🇹🇫", "🇹🇬", "🇹🇭", "🇹🇯", "🇹🇰", "🇹🇱", "🇹🇲",
|
||||
"🇹🇳", "🇹🇴", "🇹🇷", "🇹🇹", "🇹🇻", "🇹🇼", "🇹🇿", "🇺🇦", "🇺🇬", "🇺🇲", "🇺🇳", "🇺🇸",
|
||||
"🇺🇾", "🇺🇿", "🇻🇦", "🇻🇨", "🇻🇪", "🇻🇬", "🇻🇮", "🇻🇳", "🇻🇺", "🇼🇫", "🇼🇸", "🇽🇰",
|
||||
"🇾🇪", "🇾🇹", "🇿🇦", "🇿🇲", "🇿🇼", "🏴", "🏴", "🏴",
|
||||
];
|
||||
|
|
@ -1,10 +1,18 @@
|
|||
import { markdown } from "@codemirror/lang-markdown";
|
||||
import { forceParsing } from "@codemirror/language";
|
||||
import { EditorState } from "@codemirror/state";
|
||||
import { EditorView } from "@codemirror/view";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { findMarkdownTagMatches } from "@/components/MemoEditor/Editor/markdownTagRanges";
|
||||
import { tagMentionDecorations } from "@/components/MemoEditor/Editor/tagMentionDecorations";
|
||||
import { memoMarkdownExtensions } from "@/utils/memo-markdown-extension";
|
||||
|
||||
function countClass(doc: string, cls: string): number {
|
||||
const view = new EditorView({ state: EditorState.create({ doc, extensions: [tagMentionDecorations] }), parent: document.body });
|
||||
const view = new EditorView({
|
||||
state: EditorState.create({ doc, extensions: [markdown({ extensions: memoMarkdownExtensions }), tagMentionDecorations] }),
|
||||
parent: document.body,
|
||||
});
|
||||
forceParsing(view, view.state.doc.length, 1000);
|
||||
const n = view.dom.querySelectorAll(`.${cls}`).length;
|
||||
view.destroy();
|
||||
return n;
|
||||
|
|
@ -12,5 +20,199 @@ function countClass(doc: string, cls: string): number {
|
|||
|
||||
describe("tag/mention decorations", () => {
|
||||
it("decorates #tags", () => expect(countClass("a #todo and #work/sub b", "cm-memo-tag")).toBe(2));
|
||||
it("does not require a left boundary", () => expect(countClass("hello#tag 中文#标签", "cm-memo-tag")).toBe(2));
|
||||
it("uses emitted spans for ignored characters and emoji", () =>
|
||||
expect(countClass("#AB #́foo #foo/́bar #👩💻 #️⃣ ##️⃣", "cm-memo-tag")).toBe(5));
|
||||
it("keeps maximal valid prefixes and adjacent tags", () =>
|
||||
expect(countClass("#book/ #book//fiction #first#second ##tag", "cm-memo-tag")).toBe(5));
|
||||
it("does not cross formatting syntax boundaries", () => expect(countClass("#_foo_ **#urgent**", "cm-memo-tag")).toBe(1));
|
||||
it("does not decorate tags in opaque Markdown source", () =>
|
||||
expect(
|
||||
countClass("`#code` [#label](https://example.com/#link) https://example.com/#url \\#escaped #entity #ok", "cm-memo-tag"),
|
||||
).toBe(1));
|
||||
it("keeps unknown character-reference shapes in literal tag source", () =>
|
||||
expect(countClass("#R&bogus;D #Q&&bogus;D", "cm-memo-tag")).toBe(2));
|
||||
it("supplements decoded source boundaries hidden by broad parser nodes", () => {
|
||||
expect(countClass("+http://x.m/#u##v +http://x.m/#w\\|", "cm-memo-tag")).toBe(3);
|
||||
expect(countClass("#a∳b#c", "cm-memo-tag")).toBe(2);
|
||||
});
|
||||
it.each([
|
||||
".https://a.b/`#code` #tail",
|
||||
".https://a.b/[#link](/x) #tail",
|
||||
".https://a.b/ #tail",
|
||||
".https://a.b/$#math$ #tail",
|
||||
])("reparses opaque syntax swallowed by a rejected URL node: %s", (source) => expect(countClass(source, "cm-memo-tag")).toBe(1));
|
||||
it("projects emphasis closed inside a rejected URL node", () => {
|
||||
const source = "_[http://x.m/#v_>";
|
||||
const state = EditorState.create({ doc: source, extensions: [markdown({ extensions: memoMarkdownExtensions })] });
|
||||
expect(findMarkdownTagMatches(state, 0, source.length)).toEqual([{ from: 13, to: 15, source: "v", value: "v" }]);
|
||||
});
|
||||
it("coalesces literal source across a rejected URL node", () => {
|
||||
for (const [source, value] of [
|
||||
["#http://x.y/", "http"],
|
||||
["#www.x.y/", "www"],
|
||||
["#b-www.x.y/", "b-www"],
|
||||
]) {
|
||||
const state = EditorState.create({ doc: source, extensions: [markdown({ extensions: memoMarkdownExtensions })] });
|
||||
expect(findMarkdownTagMatches(state, 0, source.length).map((match) => match.value)).toEqual([value]);
|
||||
}
|
||||
});
|
||||
it("treats a rejected link reference definition as ordinary text", () => {
|
||||
expect(countClass("[#-]:(", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("[#use][bad]\n\n[bad]:(", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("[#foo@example.com]:(", "cm-memo-tag")).toBe(0);
|
||||
expect(countClass("[ https://example.com/#hidden]:(", "cm-memo-tag")).toBe(0);
|
||||
});
|
||||
it("follows CommonMark tab expansion inside blockquotes", () => {
|
||||
expect(countClass(">\t #a", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass(" >\t #a", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass(" >\t #a", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("- >\t #a", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass(">\t #a", "cm-memo-tag")).toBe(0);
|
||||
});
|
||||
it.each(["- [ ] #a", "- [x] #a", "- [ ] #a\n #b"])("decorates tags in a GFM task item: %s", (source) =>
|
||||
expect(countClass(source, "cm-memo-tag")).toBe(source.includes("#b") ? 2 : 1));
|
||||
it("keeps GFM URLs and emails opaque in task items", () =>
|
||||
expect(countClass("- [ ] https://example.com/#url #foo@example.com #tag", "cm-memo-tag")).toBe(1));
|
||||
it("keeps the exact value and range before an unknown entity semicolon", () => {
|
||||
const source = "#R&bogus;D";
|
||||
const state = EditorState.create({ doc: source, extensions: [markdown({ extensions: memoMarkdownExtensions })] });
|
||||
|
||||
expect(findMarkdownTagMatches(state, 0, source.length)).toEqual([{ from: 0, to: 8, source: "R&bogus", value: "R&bogus" }]);
|
||||
});
|
||||
it("decorates tags inside unresolved link-like source", () =>
|
||||
expect(countClass("[#tag][missing] [#other](unterminated", "cm-memo-tag")).toBe(2));
|
||||
it("decorates tags in both labels when a reference does not resolve", () => {
|
||||
expect(countClass("[x][#two]", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("[#one][#two]", "cm-memo-tag")).toBe(2);
|
||||
expect(countClass("[x][**#two**]", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("[#x][#_foo_]", "cm-memo-tag")).toBe(1);
|
||||
});
|
||||
it.each([
|
||||
"[#foo@example.com][missing]",
|
||||
"[x][#foo@example.com]",
|
||||
"[#foo@example.com](unterminated",
|
||||
"![#foo@example.com][missing]",
|
||||
"[x][_#foo@example.com_]",
|
||||
"![x][_#foo@example.com_]",
|
||||
])("keeps GFM emails opaque inside unresolved link-like source: %s", (source) => expect(countClass(source, "cm-memo-tag")).toBe(0));
|
||||
it("reparses all Markdown boundaries in unresolved second labels", () => {
|
||||
expect(countClass("[x][<#foo@example.com>] ![x][<https://example.com/#foo>]", "cm-memo-tag")).toBe(0);
|
||||
expect(countClass("[x][\\#escaped #real] ![x][#escaped #other]", "cm-memo-tag")).toBe(2);
|
||||
expect(countClass("[x][#R&D #real]", "cm-memo-tag")).toBe(2);
|
||||
expect(countClass("[x][https://example.com/#foo] ![x][https://example.com/#bar]", "cm-memo-tag")).toBe(2);
|
||||
});
|
||||
it.each(["\n", "\r", "\r\n"])("deduplicates a %j boundary in an unresolved second label", (lineEnding) =>
|
||||
expect(countClass(`[#a][${lineEnding}]${lineEnding}${lineEnding}[#a]: /x`, "cm-memo-tag")).toBe(1));
|
||||
it("keeps resolved reference and inline links opaque", () =>
|
||||
expect(countClass("[#reference][known] [#inline](/path)\n\n[known]: /path", "cm-memo-tag")).toBe(0));
|
||||
it.each([
|
||||
"[a [#link](x) b][missing]",
|
||||
"[a  b][missing]",
|
||||
])("keeps a resolved child opaque inside an unresolved outer link: %s", (source) => expect(countClass(source, "cm-memo-tag")).toBe(0));
|
||||
it("fails closed while a distant reference definition is not parsed yet", () => {
|
||||
const source = `[#hidden][known]\n\n${"x".repeat(120_000)}\n\n[known]: /path`;
|
||||
const state = EditorState.create({ doc: source, extensions: [markdown({ extensions: memoMarkdownExtensions })] });
|
||||
|
||||
expect(findMarkdownTagMatches(state, 0, 18)).toEqual([]);
|
||||
});
|
||||
it("uses CommonMark identifier normalization when resolving reference links", () => {
|
||||
expect(countClass("[#hidden][ẞ]\n\n[SS]: /path", "cm-memo-tag")).toBe(0);
|
||||
expect(countClass("[#hidden][a\0b]\n\n[a<>b]: /path", "cm-memo-tag")).toBe(0);
|
||||
expect(countClass("[#visible][\u00a0ref]\n\n[ref]: /path", "cm-memo-tag")).toBe(1);
|
||||
});
|
||||
it("follows written-GFM URL domain boundaries", () =>
|
||||
expect(
|
||||
countClass("https://example.COM/#hidden https://a_b.foo.example/路径#also-hidden https://foo_bar.example/#visible", "cm-memo-tag"),
|
||||
).toBe(1));
|
||||
it("keeps written GFM URLs with Unicode domains opaque", () =>
|
||||
expect(
|
||||
countClass(
|
||||
"https://點看.com/#hidden www.點看.com/#also-hidden https://a_b.點看.com/#still-hidden " +
|
||||
"https://foo_bar.點看/#visible https://點看.com_/#suffix hellohttps://點看.com/#joined",
|
||||
"cm-memo-tag",
|
||||
),
|
||||
).toBe(3));
|
||||
it("uses GFM protocol and www boundaries for Unicode domains", () =>
|
||||
expect(
|
||||
countClass(
|
||||
"1https://點看.com/#digit .https://點看.com/#punctuation ]www.點看.com/#www " +
|
||||
"中https://點看.com/#unicode hellohttps://點看.com/#joined",
|
||||
"cm-memo-tag",
|
||||
),
|
||||
).toBe(5));
|
||||
it("keeps URLs blocked by unbalanced brackets tag-eligible", () =>
|
||||
expect(
|
||||
countClass(
|
||||
"[https://點看.com/#protocol\n\n[www.點看.com/#www\n\n\\[https://點看.com/#escaped\n\n\\[www.點看.com/#escaped-www",
|
||||
"cm-memo-tag",
|
||||
),
|
||||
).toBe(4));
|
||||
it("keeps a URL later in unresolved bracket text opaque", () =>
|
||||
expect(countClass("[text https://點看.com/#inside\n\n[more https://example.com/#ascii", "cm-memo-tag")).toBe(0));
|
||||
it.each([
|
||||
"[ https://example.com/#hidden]",
|
||||
"[x https://example.com/#hidden]",
|
||||
"![ https://example.com/#hidden]",
|
||||
])("follows GFM URL boundaries inside unresolved bracket text: %s", (source) => expect(countClass(source, "cm-memo-tag")).toBe(0));
|
||||
it("uses one explicit separator class for written URLs", () =>
|
||||
expect(
|
||||
countClass(
|
||||
"https://點看.com/\u00a0#nbsp https://點看.com/\u2003#em-space https://點看.com/\u000b#vertical-tab " +
|
||||
"https://點看.com/\u2028#line-separator https://點看.com/\u2029#paragraph-separator https://點看.com/\u0085#nel-hidden",
|
||||
"cm-memo-tag",
|
||||
),
|
||||
).toBe(5));
|
||||
it.each([
|
||||
"`[` https://example.com/#hidden",
|
||||
"$[$ https://example.com/#hidden",
|
||||
'<span data-x="[">x</span> https://example.com/#hidden',
|
||||
"<https://x.example/[> https://example.com/#hidden",
|
||||
])("ignores brackets in earlier opaque syntax when classifying a URL: %s", (source) => expect(countClass(source, "cm-memo-tag")).toBe(0));
|
||||
it("keeps the complete suffix of a recognized GFM URL opaque", () => {
|
||||
expect(countClass("http://example.com#z", "cm-memo-tag")).toBe(0);
|
||||
expect(countClass("https://foo.example_/#visible", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("\uFEFFhttp://example.com#hidden #shown", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("before\n\n\uFEFFhttp://example.com#shown", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("\uFEFF\uFEFFhttp://example.com#shown", "cm-memo-tag")).toBe(1);
|
||||
});
|
||||
it.each([
|
||||
"#foo/bar_baz@example.com",
|
||||
"#next/item_@example.com",
|
||||
"_foo@example.com #tag_",
|
||||
])("follows complete GFM email ranges: %s", (doc) => expect(countClass(doc, "cm-memo-tag")).toBe(1));
|
||||
it("decodes escapes while reconciling repeated and adjacent GFM email ranges", () => {
|
||||
expect(countClass("#foo\\+bar@example.com #next\\_item@example.com", "cm-memo-tag")).toBe(0);
|
||||
expect(countClass("#foo\\+bar@example.com #next\\_item@example.com", "cm-memo-mention")).toBe(0);
|
||||
expect(countClass("foo@bar.com@baz.example foo@bar.com+abc@def.com", "cm-memo-mention")).toBe(0);
|
||||
});
|
||||
it.each([
|
||||
"#foo.bar@example.com",
|
||||
"#foo.bar@example.com",
|
||||
"#foo@example.com",
|
||||
])("decodes known entities before reconciling a GFM email: %s", (source) => expect(countClass(source, "cm-memo-tag")).toBe(0));
|
||||
it("keeps invalid email suffixes tag-eligible and slash-separated locals visible", () => {
|
||||
expect(countClass("#foo/bar@example.com_", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("#next/item_@example.com", "cm-memo-mention")).toBe(0);
|
||||
});
|
||||
it("does not decorate inline or block dollar math", () =>
|
||||
expect(countClass("$#inline$\n\n$$\n#block\n$$\n\n#ok", "cm-memo-tag")).toBe(1));
|
||||
it("treats dollar signs after an inline opener as math delimiters even when preceded by a backslash", () => {
|
||||
expect(countClass("$#math\\$", "cm-memo-tag")).toBe(0);
|
||||
expect(countClass("$\\$#tag$", "cm-memo-tag")).toBe(1);
|
||||
});
|
||||
it("does not retry inside an unmatched dollar run", () => expect(countClass("$$#tag$", "cm-memo-tag")).toBe(1));
|
||||
it.each(["$$\n#block", "$$meta\n#block"])("keeps unclosed flow math opaque: %s", (doc) => expect(countClass(doc, "cm-memo-tag")).toBe(0));
|
||||
it("keeps flow math scoped to its list or blockquote", () => {
|
||||
expect(countClass("> $$\n> #math\noutside #tag", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("- $$\n #math\noutside #tag", "cm-memo-tag")).toBe(1);
|
||||
});
|
||||
it("matches flow fence lengths and rejects dollar signs in metadata", () => {
|
||||
expect(countClass("$$$\n#math\n$$$\n#ok", "cm-memo-tag")).toBe(1);
|
||||
expect(countClass("$$meta$x\n#tag", "cm-memo-tag")).toBe(1);
|
||||
});
|
||||
it("allows flow math to interrupt a paragraph", () => expect(countClass("text\n$$\n#math", "cm-memo-tag")).toBe(0));
|
||||
it("does not let same-line math hide following text", () => expect(countClass("$$#inline$$\n#ok", "cm-memo-tag")).toBe(1));
|
||||
it("allows ordinary text between inline HTML tags", () => expect(countClass("<span>#inside</span>", "cm-memo-tag")).toBe(1));
|
||||
it("decorates @mentions", () => expect(countClass("hi @alice", "cm-memo-mention")).toBe(1));
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,12 +1,14 @@
|
|||
import { CompletionContext } from "@codemirror/autocomplete";
|
||||
import { markdown } from "@codemirror/lang-markdown";
|
||||
import { EditorState } from "@codemirror/state";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { makeTagCompletionSource } from "@/components/MemoEditor/Editor/tagAutocomplete";
|
||||
import { memoMarkdownExtensions } from "@/utils/memo-markdown-extension";
|
||||
|
||||
function complete(doc: string, pos: number, tags: string[]) {
|
||||
function complete(doc: string, pos: number, tags: string[], explicit = false) {
|
||||
const source = makeTagCompletionSource(() => tags);
|
||||
const state = EditorState.create({ doc });
|
||||
return source(new CompletionContext(state, pos, false));
|
||||
const state = EditorState.create({ doc, extensions: [markdown({ extensions: memoMarkdownExtensions })] });
|
||||
return source(new CompletionContext(state, pos, explicit));
|
||||
}
|
||||
|
||||
describe("tag autocomplete", () => {
|
||||
|
|
@ -22,4 +24,201 @@ describe("tag autocomplete", () => {
|
|||
it("returns null when not in a tag", () => {
|
||||
expect(complete("hello world", 11, ["todo"])).toBeNull();
|
||||
});
|
||||
|
||||
it("supports tags without a left boundary and hierarchical values", () => {
|
||||
expect(complete("hello#work/pr", 13, ["work", "work/private", "work/project"])?.options.map((option) => option.label)).toEqual([
|
||||
"work/private",
|
||||
"work/project",
|
||||
]);
|
||||
});
|
||||
|
||||
it("filters by the emitted value and replaces the complete source spelling", () => {
|
||||
const result = complete("#AB", 4, ["AB", "acorn"]);
|
||||
expect(result?.from).toBe(1);
|
||||
expect(result?.options.map((option) => option.label)).toEqual(["AB"]);
|
||||
});
|
||||
|
||||
it("keeps unknown character-reference shapes in literal tag source", () => {
|
||||
const source = "#R&bogus;D";
|
||||
const position = source.indexOf(";");
|
||||
expect(complete(source, position, ["R&bogus", "R&D"])?.options.map((option) => option.label)).toEqual(["R&bogus"]);
|
||||
});
|
||||
|
||||
it("completes tags in GFM task items", () => {
|
||||
for (const source of ["- [ ] #ta", "- [x] #ta", "- [ ] text\n #ta"]) {
|
||||
expect(complete(source, source.length, ["tag"])?.options.map((option) => option.label)).toEqual(["tag"]);
|
||||
}
|
||||
});
|
||||
|
||||
it("uses decoded boundaries hidden by a rejected URL node", () => {
|
||||
const source = "+http://x.m/#u##va";
|
||||
expect(complete(source, source.length, ["value"])?.options.map((option) => option.label)).toEqual(["value"]);
|
||||
});
|
||||
|
||||
it("does not complete in opaque syntax swallowed by a rejected URL node", () => {
|
||||
for (const source of [".https://a.b/`#code` #tail", ".https://a.b/[#link](/x) #tail", ".https://a.b/$#math$ #tail"]) {
|
||||
expect(complete(source, source.indexOf("#") + 4, ["code", "link", "math"])).toBeNull();
|
||||
expect(complete(source, source.length, ["tail"])?.options.map((option) => option.label)).toEqual(["tail"]);
|
||||
}
|
||||
});
|
||||
|
||||
it("completes tags inside unresolved link-like source", () => {
|
||||
for (const source of ["[#ta][missing]", "[#ta](unterminated"]) {
|
||||
const position = source.indexOf("]");
|
||||
expect(complete(source, position, ["tag"])?.options.map((option) => option.label)).toEqual(["tag"]);
|
||||
}
|
||||
});
|
||||
|
||||
it("completes tags inside unresolved second labels", () => {
|
||||
for (const source of ["[x][#tw]", "[#one][#tw]", "[x][**#tw**]"]) {
|
||||
const position = source.indexOf("tw") + 2;
|
||||
expect(complete(source, position, ["two"])?.options.map((option) => option.label)).toEqual(["two"]);
|
||||
}
|
||||
const emphasisBoundary = "[#x][#_foo_]";
|
||||
expect(complete(emphasisBoundary, emphasisBoundary.indexOf("foo") + 3, ["_foo_"])).toBeNull();
|
||||
});
|
||||
|
||||
it.each([
|
||||
"[#foo@example.com][missing]",
|
||||
"[x][#foo@example.com]",
|
||||
"[#foo@example.com](unterminated",
|
||||
"![#foo@example.com][missing]",
|
||||
"[x][_#foo@example.com_]",
|
||||
"![x][_#foo@example.com_]",
|
||||
])("does not complete inside a GFM email in unresolved link-like source: %s", (source) =>
|
||||
expect(complete(source, source.indexOf("foo") + 3, ["foo"])).toBeNull());
|
||||
|
||||
it("does not complete tags inside resolved references", () => {
|
||||
const source = "[#ta][known]\n\n[known]: /path";
|
||||
expect(complete(source, source.indexOf("]"), ["tag"])).toBeNull();
|
||||
});
|
||||
|
||||
it.each([
|
||||
"[a [#link](x) b][missing]",
|
||||
"[a  b][missing]",
|
||||
])("does not complete inside a resolved child of an unresolved outer link: %s", (source) =>
|
||||
expect(complete(source, source.indexOf("#") + 4, ["link", "image"])).toBeNull());
|
||||
|
||||
it("fails closed while a distant reference definition is not parsed yet", () => {
|
||||
const source = `[#ta][known]\n\n${"x".repeat(120_000)}\n\n[known]: /path`;
|
||||
expect(complete(source, source.indexOf("]"), ["tag"])).toBeNull();
|
||||
});
|
||||
|
||||
it("follows keycap candidate enumeration", () => {
|
||||
expect(complete("#️⃣", 3, ["#️⃣"])).toBeNull();
|
||||
const result = complete("##️⃣", 4, ["#️⃣"]);
|
||||
expect(result?.from).toBe(1);
|
||||
expect(result?.options.map((option) => option.label)).toEqual(["#️⃣"]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["`#to`", 4],
|
||||
["[#to](https://example.com)", 4],
|
||||
["https://example.com/#to", 23],
|
||||
["\\#to", 4],
|
||||
["#to", 7],
|
||||
["$#to$", 4],
|
||||
["$#to\\$", 4],
|
||||
["$$meta\n#to", 10],
|
||||
])("returns null in opaque Markdown source: %s", (doc, position) => expect(complete(doc, position, ["todo"])).toBeNull());
|
||||
|
||||
it("matches escaped-dollar and mismatched-run math boundaries", () => {
|
||||
expect(complete("$\\$#ta$", 6, ["tag"])?.options.map((option) => option.label)).toEqual(["tag"]);
|
||||
expect(complete("$$#ta$", 5, ["tag"])?.options.map((option) => option.label)).toEqual(["tag"]);
|
||||
});
|
||||
|
||||
it("keeps recognized URL suffixes opaque without hiding invalid-domain tags", () => {
|
||||
const url = "http://example.com#z";
|
||||
const bomURL = "\uFEFFhttp://example.com#z";
|
||||
const doubleBOMURL = "\uFEFF\uFEFFhttp://example.com#z";
|
||||
const midDocumentBOMURL = "before\n\n\uFEFFhttp://example.com#z";
|
||||
const invalidDomain = "https://foo.example_/#vi";
|
||||
|
||||
expect(complete(url, url.length, ["zebra"])).toBeNull();
|
||||
expect(complete(bomURL, bomURL.length, ["zebra"])).toBeNull();
|
||||
expect(complete(doubleBOMURL, doubleBOMURL.length, ["zebra"])?.options.map((option) => option.label)).toEqual(["zebra"]);
|
||||
expect(complete(midDocumentBOMURL, midDocumentBOMURL.length, ["zebra"])?.options.map((option) => option.label)).toEqual(["zebra"]);
|
||||
expect(complete(invalidDomain, invalidDomain.length, ["visible"])?.options.map((option) => option.label)).toEqual(["visible"]);
|
||||
});
|
||||
|
||||
it("keeps recognized Unicode-domain URL suffixes opaque", () => {
|
||||
const protocolURL = "https://點看.com/#work";
|
||||
const wwwURL = "www.點看.com/#work";
|
||||
const validPrefixedSegment = "https://a_b.點看.com/#work";
|
||||
const invalidPenultimateSegment = "https://foo_bar.點看/#work";
|
||||
const missingBoundary = "hellohttps://點看.com/#work";
|
||||
|
||||
expect(complete(protocolURL, protocolURL.length, ["work"])).toBeNull();
|
||||
expect(complete(wwwURL, wwwURL.length, ["work"])).toBeNull();
|
||||
expect(complete(validPrefixedSegment, validPrefixedSegment.length, ["work"])).toBeNull();
|
||||
expect(complete(invalidPenultimateSegment, invalidPenultimateSegment.length, ["work"])?.options.map((option) => option.label)).toEqual([
|
||||
"work",
|
||||
]);
|
||||
expect(complete(missingBoundary, missingBoundary.length, ["work"])?.options.map((option) => option.label)).toEqual(["work"]);
|
||||
});
|
||||
|
||||
it("uses GFM protocol and www boundaries for Unicode-domain URLs", () => {
|
||||
for (const url of ["1https://點看.com/#work", ".https://點看.com/#work", "]www.點看.com/#work", "中https://點看.com/#work"]) {
|
||||
expect(complete(url, url.length, ["work"])?.options.map((option) => option.label)).toEqual(["work"]);
|
||||
}
|
||||
|
||||
const joined = "hellohttps://點看.com/#work";
|
||||
expect(complete(joined, joined.length, ["work"])?.options.map((option) => option.label)).toEqual(["work"]);
|
||||
});
|
||||
|
||||
it("keeps URLs blocked by unbalanced brackets tag-eligible", () => {
|
||||
for (const url of ["[https://點看.com/#work", "[www.點看.com/#work"]) {
|
||||
expect(complete(url, url.length, ["work"])?.options.map((option) => option.label)).toEqual(["work"]);
|
||||
}
|
||||
for (const url of ["\\[https://點看.com/#work", "\\[www.點看.com/#work"]) {
|
||||
expect(complete(url, url.length, ["work"])?.options.map((option) => option.label)).toEqual(["work"]);
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
"[ https://example.com/#hidden]",
|
||||
"[x https://example.com/#hidden]",
|
||||
"![ https://example.com/#hidden]",
|
||||
])("does not complete in a GFM URL inside unresolved bracket text: %s", (source) =>
|
||||
expect(complete(source, source.indexOf("hidden") + "hidden".length, ["hidden"])).toBeNull());
|
||||
|
||||
it("uses one explicit separator class for written URLs", () => {
|
||||
for (const source of [
|
||||
"https://點看.com/\u00a0#work",
|
||||
"https://點看.com/\u2003#work",
|
||||
"https://點看.com/\u000b#work",
|
||||
"https://點看.com/\u2028#work",
|
||||
"https://點看.com/\u2029#work",
|
||||
]) {
|
||||
expect(complete(source, source.length, ["work"])?.options.map((option) => option.label)).toEqual(["work"]);
|
||||
}
|
||||
const nelURL = "https://點看.com/\u0085#work";
|
||||
expect(complete(nelURL, nelURL.length, ["work"])).toBeNull();
|
||||
});
|
||||
|
||||
it("uses reconciled GFM email ranges for completion", () => {
|
||||
const escapedPlus = "#foo\\+bar@example.com";
|
||||
const escapedUnderscore = "#next\\_item@example.com";
|
||||
const invalidSuffix = "#foo/bar@example.com_";
|
||||
|
||||
expect(complete(escapedPlus, 4, ["foo"])).toBeNull();
|
||||
expect(complete(escapedUnderscore, 5, ["next"])).toBeNull();
|
||||
expect(complete(invalidSuffix, 8, ["foo/bar"])?.options.map((option) => option.label)).toEqual(["foo/bar"]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
"#foo.bar@example.com",
|
||||
"#foo.bar@example.com",
|
||||
"#foo@example.com",
|
||||
])("does not complete inside a GFM email joined by a known entity: %s", (source) => expect(complete(source, 4, ["foo"])).toBeNull());
|
||||
|
||||
it("offers all tags on explicitly completed bare introducer", () => {
|
||||
expect(complete("hello #", 7, ["todo", "work"], true)?.options.map((option) => option.label)).toEqual(["todo", "work"]);
|
||||
});
|
||||
|
||||
it("does not treat a keycap emoji hash as an explicit bare introducer", () => {
|
||||
expect(complete("#️⃣", 1, ["todo"], true)).toBeNull();
|
||||
expect(complete("##️⃣", 2, ["todo"], true)).toBeNull();
|
||||
expect(complete("#first#️⃣", 7, ["todo"], true)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
|
|
|||
42
web/tests/gfm-email.test.ts
Normal file
42
web/tests/gfm-email.test.ts
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
import { parser as markdownParser } from "@lezer/markdown";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { findGFMEmailMatches, findMarkdownGFMEmailRanges, type MarkdownSourceNode } from "@/utils/gfm-email";
|
||||
import { resolvedMarkdownLinkRanges } from "@/utils/markdown-link";
|
||||
import { memoMarkdownExtensions } from "@/utils/memo-markdown-extension";
|
||||
|
||||
describe("GFM email source ranges", () => {
|
||||
const range = (from: number, to: number, value: string) => ({ from, to, value });
|
||||
const sourceParser = markdownParser.configure(memoMarkdownExtensions);
|
||||
|
||||
it.each([
|
||||
["#foo/bar_baz@example.com", [range(5, 24, "bar_baz@example.com")]],
|
||||
["#foo\\+bar@example.com", [range(1, 21, "foo+bar@example.com")]],
|
||||
["#next\\_item@example.com", [range(1, 23, "next_item@example.com")]],
|
||||
["#foo.bar@example.com", [range(1, 24, "foo.bar@example.com")]],
|
||||
["#foo.bar@example.com", [range(1, 27, "foo.bar@example.com")]],
|
||||
["#foo@example.com", [range(1, 20, "foo@example.com")]],
|
||||
["#foo&bogus;bar@example.com", [range(11, 26, "bar@example.com")]],
|
||||
["#foo!bar@example.com", [range(5, 20, "bar@example.com")]],
|
||||
["foo@bar.com@baz.example", [range(0, 11, "foo@bar.com")]],
|
||||
["foo@bar.com+abc@def.com", [range(0, 11, "foo@bar.com"), range(11, 23, "+abc@def.com")]],
|
||||
["mail@example.com.", [range(0, 16, "mail@example.com")]],
|
||||
["mail@example.com_", []],
|
||||
["mail@example.com-", []],
|
||||
["mail@example.c_m", [range(0, 16, "mail@example.c_m")]],
|
||||
["mail@example.c_", []],
|
||||
["😀foo@example.com", [range(2, 17, "foo@example.com")]],
|
||||
["(mail@example.com", [range(1, 17, "mail@example.com")]],
|
||||
])("matches canonical ranges in %s", (source, expected) => {
|
||||
expect(findGFMEmailMatches(source)).toEqual(expected);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["[x][_#foo@example.com_]", [range(6, 21, "foo@example.com")]],
|
||||
["![x][_#foo@example.com_]", [range(7, 22, "foo@example.com")]],
|
||||
["[x][`#foo@example.com`]", []],
|
||||
])("reparses unresolved second-label Markdown in %s", (source, expected) => {
|
||||
const root = sourceParser.parse(source).topNode as MarkdownSourceNode;
|
||||
|
||||
expect(findMarkdownGFMEmailRanges(source, root, resolvedMarkdownLinkRanges(source, root), true)).toEqual(expected);
|
||||
});
|
||||
});
|
||||
21
web/tests/gfm-url.test.ts
Normal file
21
web/tests/gfm-url.test.ts
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { writtenGFMURLSourceRange } from "@/utils/gfm-url";
|
||||
|
||||
describe("written GFM URL source ranges", () => {
|
||||
it.each([
|
||||
["www.example.com/path?!", "www.example.com/path"],
|
||||
['www.example.com/path\"', "www.example.com/path"],
|
||||
["www.example.com/path))", "www.example.com/path"],
|
||||
["www.example.com/path(value))", "www.example.com/path(value)"],
|
||||
["www.example.com/path&", "www.example.com/path"],
|
||||
["www.example.com/path]", "www.example.com/path"],
|
||||
["www.example.com/path(value)", "www.example.com/path(value)"],
|
||||
])("trims the canonical tail in %s", (source, expected) => {
|
||||
expect(writtenGFMURLSourceRange(source, 0)).toEqual({ from: 0, to: expected.length });
|
||||
});
|
||||
|
||||
it("keeps punctuation that is followed by URL content", () => {
|
||||
const source = "www.example.com/path?#tag";
|
||||
expect(writtenGFMURLSourceRange(source, 0)).toEqual({ from: 0, to: source.length });
|
||||
});
|
||||
});
|
||||
|
|
@ -11,4 +11,11 @@ describe("parseMentionsFromText", () => {
|
|||
const username = `a${"b".repeat(63)}`;
|
||||
expect(parseMentionsFromText(`@${username}`)).toEqual([{ type: "text", value: `@${username}` }]);
|
||||
});
|
||||
|
||||
it("keeps rejected adjacent mentions in the same literal text segment", () => {
|
||||
expect(parseMentionsFromText("@example.com@alice", "_")).toEqual([
|
||||
{ type: "mention", value: "example" },
|
||||
{ type: "text", value: ".com@alice" },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,16 +1,20 @@
|
|||
import { renderToStaticMarkup } from "react-dom/server";
|
||||
import ReactMarkdown from "react-markdown";
|
||||
import remarkBreaks from "remark-breaks";
|
||||
import remarkGfm from "remark-gfm";
|
||||
import remarkMath from "remark-math";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { remarkMention } from "@/utils/remark-plugins/remark-mention";
|
||||
import { remarkTag } from "@/utils/remark-plugins/remark-tag";
|
||||
|
||||
const renderMarkdown = (content: string): string =>
|
||||
renderToStaticMarkup(
|
||||
<ReactMarkdown remarkPlugins={[remarkGfm, remarkTag]}>
|
||||
{content}
|
||||
</ReactMarkdown>,
|
||||
<ReactMarkdown remarkPlugins={[remarkMath, remarkGfm, remarkTag, remarkMention, remarkBreaks]}>{content}</ReactMarkdown>,
|
||||
);
|
||||
|
||||
const renderMarkdownWithoutMath = (content: string): string =>
|
||||
renderToStaticMarkup(<ReactMarkdown remarkPlugins={[remarkGfm, remarkTag, remarkMention, remarkBreaks]}>{content}</ReactMarkdown>);
|
||||
|
||||
describe("remarkTag", () => {
|
||||
it("does not turn URL fragments inside autolinks into tags", () => {
|
||||
const html = renderMarkdown("https://github.com/dmtrKovalenko/fff#pi-agent-extension\n\nProject #memo-tag");
|
||||
|
|
@ -20,6 +24,317 @@ describe("remarkTag", () => {
|
|||
expect(html).toContain('data-tag="memo-tag"');
|
||||
});
|
||||
|
||||
it("uses written GFM URL nodes as the opaque boundary", () => {
|
||||
const html = renderMarkdown("https://example.com/#hidden https://localhost/#local HTTP://example.com/#upper ftp://example.com/#ftp");
|
||||
|
||||
expect(html).not.toContain('data-tag="hidden"');
|
||||
expect(html).toContain('data-tag="local"');
|
||||
expect(html).toContain('data-tag="upper"');
|
||||
expect(html).toContain('data-tag="ftp"');
|
||||
});
|
||||
|
||||
it("keeps complete written-GFM link suffixes opaque", () => {
|
||||
const html = renderMarkdown("x https://example.com/%#&_; http://example.com#z\n\nhttps://foo.example_/#suffix #ok");
|
||||
|
||||
expect(html).not.toContain('data-tag="&"');
|
||||
expect(html).not.toContain('data-tag="z"');
|
||||
expect(html).toContain('data-tag="suffix"');
|
||||
expect(html).toContain('data-tag="ok"');
|
||||
});
|
||||
|
||||
it("coalesces text after unwrapping a remark-only URL", () => {
|
||||
const html = renderMarkdown("#&HTTP://example.com/");
|
||||
|
||||
expect(html).toContain('<span class="tag" data-tag="&HTTP">#&HTTP</span>://example.com/');
|
||||
expect(html).not.toContain('data-tag="&"');
|
||||
});
|
||||
|
||||
it("recovers positionless text around remark-only email links", () => {
|
||||
const html = renderMarkdown("\\+foo@example.com #ok *\\+foo@example.com #inside*");
|
||||
const fallbackHTML = renderMarkdownWithoutMath("$\\+foo@example.com #math$ #outside");
|
||||
|
||||
expect(html).toContain('data-tag="ok"');
|
||||
expect(html).toContain('data-tag="inside"');
|
||||
expect(fallbackHTML).not.toContain('data-tag="math"');
|
||||
expect(fallbackHTML).toContain('data-tag="outside"');
|
||||
});
|
||||
|
||||
it("maps decoded positionless siblings without stealing a later source match", () => {
|
||||
const html = renderMarkdown("#same \\+foo@example.com #same #left& \\+bar@example.com #right");
|
||||
|
||||
expect(html).toContain('#same <a href="mailto:+foo@example.com">+foo@example.com</a> <span class="tag" data-tag="same">#same</span>');
|
||||
expect(html).toContain('data-tag="left"');
|
||||
expect(html).toContain('data-tag="right"');
|
||||
expect(html.match(/data-tag="same"/g)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it.each([
|
||||
"*\\+foo@example.com #ok*",
|
||||
"# \\+foo@example.com #ok",
|
||||
"~~\\+foo@example.com #ok~~",
|
||||
"| value |\n| - |\n| \\+foo@example.com #ok |",
|
||||
])("recovers positionless text inside a transparent Markdown container: %s", (source) => {
|
||||
expect(renderMarkdown(source)).toContain('data-tag="ok"');
|
||||
});
|
||||
|
||||
it("keeps complete entity and escape ranges as source boundaries", () => {
|
||||
const html = renderMarkdown("https://localhost/#w# HTTP://example.com/#q\\+ HTTP://example.com/#r&x");
|
||||
|
||||
expect(html).toContain('data-tag="w"');
|
||||
expect(html).not.toContain('data-tag="w&"');
|
||||
expect(html).not.toContain('data-tag="35"');
|
||||
expect(html).toContain('data-tag="q"');
|
||||
expect(html).not.toContain('data-tag="q+"');
|
||||
expect(html).toContain('data-tag="r"');
|
||||
expect(html).not.toContain('data-tag="r&"');
|
||||
expect(html).not.toContain("\\+");
|
||||
expect(html).not.toContain("&amp;");
|
||||
});
|
||||
it("supplements decoded boundaries hidden by broad parser nodes", () => {
|
||||
const rejectedURL = renderMarkdown("+http://x.m/#u##v +http://x.m/#w\\|");
|
||||
expect(rejectedURL).toContain('data-tag="u"');
|
||||
expect(rejectedURL).toContain('data-tag="v"');
|
||||
expect(rejectedURL).toContain('data-tag="w"');
|
||||
expect(rejectedURL).not.toContain('data-tag="35"');
|
||||
|
||||
const longEntity = renderMarkdown("#a∳b#c");
|
||||
expect(longEntity).toContain('data-tag="a"');
|
||||
expect(longEntity).toContain('data-tag="c"');
|
||||
});
|
||||
it.each(["~#a!&&)*", ":#b[&-&)", "&\\;#-"])("does not widen decoded source boundaries: %s", (source) => {
|
||||
expect(renderMarkdown(source)).toContain("data-tag=");
|
||||
});
|
||||
it("keeps tags after parser-omitted trailing whitespace", () => {
|
||||
expect(renderMarkdown("before \n#tag")).toContain('data-tag="tag"');
|
||||
expect(renderMarkdown(";.$;\nab- )> \n#&")).toContain('data-tag="&"');
|
||||
});
|
||||
it("does not resolve a link through a rejected reference definition", () => {
|
||||
const html = renderMarkdown("[#use][bad]\n\n[bad]:(");
|
||||
expect(html).toContain('data-tag="use"');
|
||||
expect(html).not.toContain("<a ");
|
||||
});
|
||||
it("keeps GFM autolinks inside a rejected reference definition opaque", () => {
|
||||
expect(renderMarkdown("[#foo@example.com]:(")).not.toContain('data-tag="foo"');
|
||||
expect(renderMarkdown("[ https://example.com/#hidden]:(")).not.toContain('data-tag="hidden"');
|
||||
});
|
||||
it.each([
|
||||
".https://a.b/`#code` #tail",
|
||||
".https://a.b/[#link](/x) #tail",
|
||||
".https://a.b/ #tail",
|
||||
".https://a.b/$#math$ #tail",
|
||||
])("reparses opaque syntax swallowed by a rejected URL node: %s", (source) => {
|
||||
const html = renderMarkdown(source);
|
||||
expect(html).not.toContain('data-tag="code"');
|
||||
expect(html).not.toContain('data-tag="link"');
|
||||
expect(html).not.toContain('data-tag="image"');
|
||||
expect(html).not.toContain('data-tag="math"');
|
||||
expect(html).toContain('data-tag="tail"');
|
||||
});
|
||||
it("projects emphasis closed inside a rejected URL node", () => {
|
||||
const html = renderMarkdown("_[http://x.m/#v_>");
|
||||
expect(html).toContain('data-tag="v"');
|
||||
expect(html).not.toContain('data-tag="v_"');
|
||||
});
|
||||
|
||||
it("follows written-GFM domain and email boundaries", () => {
|
||||
const html = renderMarkdown(
|
||||
"https://example.COM/#hidden www.example.123/#also-hidden https://a_b.foo.example/路径#still-hidden " +
|
||||
"https://foo_bar.example/#visible foo#mail@example.com <foo#angle@example.com> foo@example.com/#after",
|
||||
);
|
||||
|
||||
expect(html).not.toContain('data-tag="hidden"');
|
||||
expect(html).not.toContain('data-tag="also-hidden"');
|
||||
expect(html).not.toContain('data-tag="still-hidden"');
|
||||
expect(html).not.toContain('data-tag="angle"');
|
||||
expect(html).toContain('data-tag="visible"');
|
||||
expect(html).not.toContain('data-tag="mail"');
|
||||
expect(html).toContain('foo#<a href="mailto:mail@example.com">mail@example.com</a>');
|
||||
expect(html).toContain('data-tag="after"');
|
||||
});
|
||||
|
||||
it("keeps written GFM URLs with Unicode domains opaque", () => {
|
||||
const html = renderMarkdown(
|
||||
"https://點看.com/#hidden www.點看.com/#also-hidden https://a_b.點看.com/#still-hidden " +
|
||||
"https://foo_bar.點看/#visible https://點看.com_/#suffix hellohttps://點看.com/#joined",
|
||||
);
|
||||
|
||||
expect(html).toContain('href="https://%E9%BB%9E%E7%9C%8B.com/#hidden"');
|
||||
expect(html).toContain('href="http://www.%E9%BB%9E%E7%9C%8B.com/#also-hidden"');
|
||||
expect(html).not.toContain('data-tag="hidden"');
|
||||
expect(html).not.toContain('data-tag="also-hidden"');
|
||||
expect(html).not.toContain('data-tag="still-hidden"');
|
||||
expect(html).toContain('data-tag="visible"');
|
||||
expect(html).toContain('data-tag="suffix"');
|
||||
expect(html).toContain('data-tag="joined"');
|
||||
});
|
||||
|
||||
it("uses GFM protocol and www boundaries for Unicode-domain URLs", () => {
|
||||
const html = renderMarkdown(
|
||||
"1https://點看.com/#digit .https://點看.com/#punctuation ]www.點看.com/#www " +
|
||||
"中https://點看.com/#unicode hellohttps://點看.com/#joined",
|
||||
);
|
||||
|
||||
expect(html).toContain('data-tag="digit"');
|
||||
expect(html).toContain('data-tag="punctuation"');
|
||||
expect(html).toContain('data-tag="www"');
|
||||
expect(html).toContain('data-tag="unicode"');
|
||||
expect(html).toContain('data-tag="joined"');
|
||||
});
|
||||
|
||||
it("keeps URLs blocked by unbalanced brackets tag-eligible", () => {
|
||||
const html = renderMarkdown(
|
||||
"[https://點看.com/#protocol\n\n[www.點看.com/#www\n\n\\[https://點看.com/#escaped\n\n\\[www.點看.com/#escaped-www",
|
||||
);
|
||||
|
||||
expect(html).toContain('data-tag="protocol"');
|
||||
expect(html).toContain('data-tag="www"');
|
||||
expect(html).toContain('data-tag="escaped"');
|
||||
expect(html).toContain('data-tag="escaped-www"');
|
||||
});
|
||||
|
||||
it("keeps a URL later in unresolved bracket text opaque", () => {
|
||||
const html = renderMarkdown("[text https://點看.com/#inside\n\n[more https://example.com/#ascii");
|
||||
|
||||
expect(html).not.toContain('data-tag="inside"');
|
||||
expect(html).not.toContain('data-tag="ascii"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
"[ https://example.com/#hidden]",
|
||||
"[x https://example.com/#hidden]",
|
||||
"![ https://example.com/#hidden]",
|
||||
])("follows GFM URL boundaries inside unresolved bracket text: %s", (source) =>
|
||||
expect(renderMarkdown(source)).not.toContain('data-tag="hidden"'));
|
||||
|
||||
it("uses one explicit separator class for written URLs", () => {
|
||||
const html = renderMarkdown(
|
||||
"https://點看.com/\u00a0#nbsp https://點看.com/\u2003#em-space https://點看.com/\u000b#vertical-tab " +
|
||||
"https://點看.com/\u2028#line-separator https://點看.com/\u2029#paragraph-separator https://點看.com/\u0085#nel-hidden",
|
||||
);
|
||||
|
||||
expect(html).toContain('data-tag="nbsp"');
|
||||
expect(html).toContain('data-tag="em-space"');
|
||||
expect(html).toContain('data-tag="vertical-tab"');
|
||||
expect(html).toContain('data-tag="line-separator"');
|
||||
expect(html).toContain('data-tag="paragraph-separator"');
|
||||
expect(html).not.toContain('data-tag="nel-hidden"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
"`[` https://example.com/#hidden",
|
||||
"$[$ https://example.com/#hidden",
|
||||
'<span data-x="[">x</span> https://example.com/#hidden',
|
||||
"<https://x.example/[> https://example.com/#hidden",
|
||||
])("ignores brackets in earlier opaque syntax when classifying a URL: %s", (source) => {
|
||||
expect(renderMarkdown(source)).not.toContain('data-tag="hidden"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"#foo/bar_baz@example.com",
|
||||
"foo",
|
||||
"foo/bar_baz",
|
||||
'<span class="tag" data-tag="foo">#foo</span>/<a href="mailto:bar_baz@example.com">bar_baz@example.com</a>',
|
||||
],
|
||||
[
|
||||
"#next/item_@example.com",
|
||||
"next",
|
||||
"next/item_",
|
||||
'<span class="tag" data-tag="next">#next</span>/<a href="mailto:item_@example.com">item_@example.com</a>',
|
||||
],
|
||||
[
|
||||
"_foo@example.com #tag_",
|
||||
"tag",
|
||||
"tag_",
|
||||
'<em><a href="mailto:foo@example.com">foo@example.com</a> <span class="tag" data-tag="tag">#tag</span></em>',
|
||||
],
|
||||
])("uses the complete GFM email and emphasis source ranges: %s", (source, tag, excludedTag, rendered) => {
|
||||
const html = renderMarkdown(source);
|
||||
|
||||
expect(html).toContain(`data-tag="${tag}"`);
|
||||
expect(html).not.toContain(`data-tag="${excludedTag}"`);
|
||||
expect(html).toContain(rendered);
|
||||
});
|
||||
|
||||
it("keeps only canonical GFM emails opaque before the mention transform", () => {
|
||||
const html = renderMarkdown("#foo/bar_baz@example.com @alice\n\n#next/item_@example.com @bob\n\n_foo@example.com #tag_ @carol");
|
||||
|
||||
expect(html).not.toContain('data-mention="example"');
|
||||
expect(html).toContain('data-mention="alice"');
|
||||
expect(html).toContain('data-mention="bob"');
|
||||
expect(html).toContain('data-mention="carol"');
|
||||
expect(html).toContain('<em><a href="mailto:foo@example.com">foo@example.com</a> <span class="tag" data-tag="tag">#tag</span></em>');
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"__foo@example.com #tag__ @carol",
|
||||
'<strong><a href="mailto:foo@example.com">foo@example.com</a> <span class="tag" data-tag="tag">#tag</span></strong>',
|
||||
],
|
||||
[
|
||||
"___foo@example.com #tag___ @carol",
|
||||
'<em><strong><a href="mailto:foo@example.com">foo@example.com</a> <span class="tag" data-tag="tag">#tag</span></strong></em>',
|
||||
],
|
||||
[
|
||||
"____foo@example.com #tag____ @carol",
|
||||
'<strong><strong><a href="mailto:foo@example.com">foo@example.com</a> <span class="tag" data-tag="tag">#tag</span></strong></strong>',
|
||||
],
|
||||
])("reconciles nested underscore formatting around GFM emails: %s", (source, expected) => {
|
||||
const html = renderMarkdown(source);
|
||||
|
||||
expect(html).toContain(expected);
|
||||
expect(html).toContain('data-mention="carol"');
|
||||
});
|
||||
|
||||
it("decodes escapes while reconciling repeated and adjacent GFM email ranges", () => {
|
||||
const escaped = renderMarkdown("#foo\\+bar@example.com #next\\_item@example.com @alice");
|
||||
const repeated = renderMarkdown("foo@bar.com@baz.example");
|
||||
const adjacent = renderMarkdown("foo@bar.com+abc@def.com");
|
||||
const invalidSuffix = renderMarkdown("#foo/bar@example.com_");
|
||||
|
||||
expect(escaped).toContain('#<a href="mailto:foo+bar@example.com">foo+bar@example.com</a>');
|
||||
expect(escaped).toContain('#<a href="mailto:next_item@example.com">next_item@example.com</a>');
|
||||
expect(escaped).not.toContain('data-tag="foo"');
|
||||
expect(escaped).not.toContain('data-tag="next"');
|
||||
expect(escaped).not.toContain('data-mention="example"');
|
||||
expect(escaped).toContain('data-mention="alice"');
|
||||
expect(repeated).toContain('<a href="mailto:foo@bar.com">foo@bar.com</a>@baz.example');
|
||||
expect(repeated.match(/href="mailto:/g)).toHaveLength(1);
|
||||
expect(adjacent).toContain('<a href="mailto:foo@bar.com">foo@bar.com</a><a href="mailto:+abc@def.com">+abc@def.com</a>');
|
||||
expect(adjacent.match(/href="mailto:/g)).toHaveLength(2);
|
||||
expect(invalidSuffix).toContain('data-tag="foo/bar"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
["#foo.bar@example.com", "foo.bar@example.com"],
|
||||
["#foo.bar@example.com", "foo.bar@example.com"],
|
||||
["#foo@example.com", "foo@example.com"],
|
||||
])("decodes known entities before reconciling a GFM email: %s", (source, address) => {
|
||||
const html = renderMarkdown(source);
|
||||
|
||||
expect(html).not.toContain('data-tag="foo"');
|
||||
expect(html).toContain(`href="mailto:${address}"`);
|
||||
});
|
||||
|
||||
it("uses only contiguous literal source across mention node boundaries", () => {
|
||||
const html = renderMarkdown("#next/item_@example.com@alice");
|
||||
const spacedHTML = renderMarkdown("#next/item_@example.com @alice");
|
||||
const linkedHTML = renderMarkdown("[mail](https://example.com)@alice");
|
||||
const strongHTML = renderMarkdown("**bold**@alice");
|
||||
const emphasisHTML = renderMarkdown("*x*@alice");
|
||||
const tagHTML = renderMarkdown("#tag@alice");
|
||||
const adjacentMentionsHTML = renderMarkdown("@alice@bob");
|
||||
|
||||
expect(html).not.toContain('data-mention="alice"');
|
||||
expect(spacedHTML).toContain('data-mention="alice"');
|
||||
expect(linkedHTML).toContain('data-mention="alice"');
|
||||
expect(strongHTML).toContain('data-mention="alice"');
|
||||
expect(emphasisHTML).toContain('data-mention="alice"');
|
||||
expect(tagHTML).not.toContain('data-mention="alice"');
|
||||
expect(adjacentMentionsHTML.match(/data-mention=/g)).toHaveLength(1);
|
||||
expect(adjacentMentionsHTML).not.toContain('data-mention="bob"');
|
||||
});
|
||||
|
||||
it("does not turn link text or reference link fragments into tags", () => {
|
||||
const html = renderMarkdown(
|
||||
[
|
||||
|
|
@ -45,11 +360,104 @@ describe("remarkTag", () => {
|
|||
expect(html).toContain('data-tag="memo-tag"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
"[a [#link](x) b][missing]",
|
||||
"[a  b][missing]",
|
||||
])("keeps a resolved child opaque inside an unresolved outer link: %s", (source) =>
|
||||
expect(renderMarkdown(source)).not.toMatch(/data-tag="(?:link|image)"/));
|
||||
|
||||
it("treats unresolved link-like syntax as ordinary tag-eligible text", () => {
|
||||
const html = renderMarkdown("[#tag][missing] [#other](unterminated");
|
||||
|
||||
expect(html).toContain('data-tag="tag"');
|
||||
expect(html).toContain('data-tag="other"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
"[#foo@example.com][missing]",
|
||||
"[x][#foo@example.com]",
|
||||
"[#foo@example.com](unterminated",
|
||||
"![#foo@example.com][missing]",
|
||||
"[x][_#foo@example.com_]",
|
||||
"![x][_#foo@example.com_]",
|
||||
])("keeps GFM emails opaque inside unresolved link-like source: %s", (source) => {
|
||||
const html = renderMarkdown(source);
|
||||
|
||||
expect(html).not.toContain('data-tag="foo"');
|
||||
expect(html).toContain('href="mailto:foo@example.com"');
|
||||
});
|
||||
|
||||
it("keeps code opaque while reparsing an unresolved second label", () => {
|
||||
const html = renderMarkdown("[x][`#foo@example.com`]");
|
||||
|
||||
expect(html).not.toContain('data-tag="foo"');
|
||||
expect(html).not.toContain('href="mailto:foo@example.com"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
"[x][https://example.com/#foo]",
|
||||
"![x][https://example.com/#foo]",
|
||||
])("does not give a second-label URL a synthetic boundary: %s", (source) => expect(renderMarkdown(source)).toContain('data-tag="foo"'));
|
||||
|
||||
it.each(["\n", "\r", "\r\n"])("deduplicates a %j boundary projected from an unresolved second label", (lineEnding) => {
|
||||
const html = renderMarkdown(`[#a][${lineEnding}]${lineEnding}${lineEnding}[#a]: /x`);
|
||||
|
||||
expect(html).toContain('data-tag="a"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
"[x][<#foo@example.com>]",
|
||||
"![x][<#foo@example.com>]",
|
||||
"[x][<https://example.com/#foo>]",
|
||||
"![x][<https://example.com/#foo>]",
|
||||
])("keeps autolinks opaque while reparsing an unresolved second label: %s", (source) =>
|
||||
expect(renderMarkdown(source)).not.toContain('data-tag="foo"'));
|
||||
|
||||
it.each([
|
||||
"[x][\\#escaped #real]",
|
||||
"![x][\\#escaped #real]",
|
||||
"[x][#escaped #real]",
|
||||
"![x][#escaped #real]",
|
||||
])("projects escaped introducer boundaries from an unresolved second label: %s", (source) => {
|
||||
const html = renderMarkdown(source);
|
||||
|
||||
expect(html).not.toContain('data-tag="escaped"');
|
||||
expect(html).toContain('data-tag="real"');
|
||||
});
|
||||
|
||||
it.each(["[x][#R&D #real]", "![x][#R&D #real]"])("projects entity boundaries from an unresolved second label: %s", (source) => {
|
||||
const html = renderMarkdown(source);
|
||||
|
||||
expect(html).toContain('data-tag="R"');
|
||||
expect(html).toContain('data-tag="real"');
|
||||
expect(html).not.toContain('data-tag="R&D"');
|
||||
});
|
||||
|
||||
it("recognizes tags in unresolved second labels while preserving emphasis boundaries", () => {
|
||||
const html = renderMarkdown("[x][#two] [#one][#two] [x][**#two**] [#x][#_foo_]");
|
||||
|
||||
expect(html.match(/data-tag="two"/g)).toHaveLength(3);
|
||||
expect(html).toContain('data-tag="one"');
|
||||
expect(html).toContain('data-tag="x"');
|
||||
expect(html).not.toContain('data-tag="_foo_"');
|
||||
});
|
||||
|
||||
it("uses CommonMark identifier normalization for reference links", () => {
|
||||
const sharpS = renderMarkdown("[#hidden][ẞ]\n\n[SS]: /path");
|
||||
const nul = renderMarkdown("[#hidden][a\0b]\n\n[a<>b]: /path");
|
||||
const nonbreakingSpace = renderMarkdown("[#visible][\u00a0ref]\n\n[ref]: /path");
|
||||
|
||||
expect(sharpS).not.toContain('data-tag="hidden"');
|
||||
expect(nul).not.toContain('data-tag="hidden"');
|
||||
expect(nonbreakingSpace).toContain('data-tag="visible"');
|
||||
});
|
||||
|
||||
it("continues to turn formatted text outside links into tags", () => {
|
||||
const html = renderMarkdown("**#urgent** and _#later_");
|
||||
const html = renderMarkdown("**#urgent** and _#later_ but not #_split_");
|
||||
|
||||
expect(html).toContain('data-tag="urgent"');
|
||||
expect(html).toContain('data-tag="later"');
|
||||
expect(html).not.toContain('data-tag="_split_"');
|
||||
});
|
||||
|
||||
it("does not turn a backslash-escaped \\#tag into a tag, but still tags an unescaped one", () => {
|
||||
|
|
@ -80,11 +488,164 @@ describe("remarkTag", () => {
|
|||
});
|
||||
|
||||
it("still tags a hash that shares a text node with an entity reference", () => {
|
||||
// The source slice ("...&...") differs from the decoded value, so the
|
||||
// escape-aware path bows out and the tag is detected the original way.
|
||||
const html = renderMarkdown("Tom & Jerry #cartoon");
|
||||
|
||||
expect(html).toContain('data-tag="cartoon"');
|
||||
expect(html).toContain("Tom & Jerry");
|
||||
});
|
||||
|
||||
it("uses original-source syntax boundaries rather than decoded text", () => {
|
||||
const html = renderMarkdown("#tag #tag #foo\\+bar #R&D #R&D #R&bogus;D #Q&&bogus;D #ok");
|
||||
|
||||
expect(html).not.toContain('data-tag="tag"');
|
||||
expect(html).toContain('data-tag="foo"');
|
||||
expect(html).toContain('data-tag="R"');
|
||||
expect(html).toContain('data-tag="R&D"');
|
||||
expect(html).toContain('data-tag="R&bogus"');
|
||||
expect(html).toContain('data-tag="Q"');
|
||||
expect(html).toContain('data-tag="ok"');
|
||||
});
|
||||
|
||||
it("renders source spelling while exposing the emitted tag value", () => {
|
||||
const html = renderMarkdown("#AB #́foo #foo/́bar #👩💻");
|
||||
|
||||
expect(html).toContain('<span class="tag" data-tag="AB">#AB</span>');
|
||||
expect(html).toContain('<span class="tag" data-tag="foo">#́foo</span>');
|
||||
expect(html).toContain('<span class="tag" data-tag="foo/bar">#foo/́bar</span>');
|
||||
expect(html).toContain('data-tag="👩💻"');
|
||||
});
|
||||
|
||||
it("uses maximal-prefix hierarchy and adjacent-introducer rules", () => {
|
||||
const html = renderMarkdown("#book/ #book//fiction #first#second ##tag #️⃣ ##️⃣");
|
||||
|
||||
expect(html.match(/data-tag="book"/g)).toHaveLength(2);
|
||||
expect(html).toContain('data-tag="first"');
|
||||
expect(html).toContain('data-tag="second"');
|
||||
expect(html).toContain('data-tag="tag"');
|
||||
expect(html.match(/data-tag="#️⃣"/g)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("maps tag source on both sides of CRLF line endings", () => {
|
||||
const html = renderMarkdown("#one\r\n#two");
|
||||
|
||||
expect(html).toContain('data-tag="one"');
|
||||
expect(html).toContain('data-tag="two"');
|
||||
});
|
||||
|
||||
it.each([1, 2, 3])("maps paragraph continuation indentation of %d spaces", (indent) => {
|
||||
const spaces = " ".repeat(indent);
|
||||
const after = renderMarkdown(`#before\n${spaces}text`);
|
||||
const before = renderMarkdown(`text\n${spaces}#after`);
|
||||
|
||||
expect(after).toContain('data-tag="before"');
|
||||
expect(before).toContain('data-tag="after"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
["- #before\n text", "before"],
|
||||
["- text\n #after", "after"],
|
||||
["1. #before\n text", "before"],
|
||||
["1. text\n #after", "after"],
|
||||
["> #before\n> text", "before"],
|
||||
["> text\n> #after", "after"],
|
||||
["#before\n\t#after", "before"],
|
||||
["#before\n\t#after", "after"],
|
||||
["- > #before\n > #after", "before"],
|
||||
["- > #before\n > #after", "after"],
|
||||
["> - #before\n> #after", "before"],
|
||||
["> - #before\n> #after", "after"],
|
||||
])("maps parser-consumed continuation prefixes in %s", (source, tag) => {
|
||||
expect(renderMarkdown(source)).toContain(`data-tag="${tag}"`);
|
||||
});
|
||||
|
||||
it("maps CommonMark NUL replacement without losing adjacent tags", () => {
|
||||
const html = renderMarkdown("\uFEFF#bom \0#left #right\0");
|
||||
|
||||
expect(html).toContain('<span class="tag" data-tag="bom">#bom</span>');
|
||||
expect(html).not.toContain('data-tag="bo"');
|
||||
expect(html).toContain('data-tag="left"');
|
||||
expect(html).toContain('data-tag="right"');
|
||||
});
|
||||
|
||||
it("keeps URL context aligned after a leading BOM", () => {
|
||||
const html = renderMarkdown("\uFEFFhttp://example.com#hidden #shown");
|
||||
|
||||
expect(html).not.toContain('data-tag="hidden"');
|
||||
expect(html).toContain('data-tag="shown"');
|
||||
});
|
||||
|
||||
it("treats a second leading BOM as ordinary text", () => {
|
||||
const html = renderMarkdown("\uFEFF\uFEFFhttp://example.com#shown");
|
||||
|
||||
expect(html).toContain('data-tag="shown"');
|
||||
});
|
||||
|
||||
it("does not treat a mid-document BOM as a written URL boundary", () => {
|
||||
const html = renderMarkdown("before\n\n\uFEFFhttp://example.com#shown");
|
||||
|
||||
expect(html).toContain('data-tag="shown"');
|
||||
});
|
||||
|
||||
it("keeps source positions before hard-break and mention transforms", () => {
|
||||
const html = renderMarkdown("first #one\nsecond #two\n#tag@alice #tag @bob");
|
||||
|
||||
expect(html).toContain('data-tag="one"');
|
||||
expect(html).toContain('data-tag="two"');
|
||||
expect(html).not.toContain('data-mention="alice"');
|
||||
expect(html).toContain('data-mention="bob"');
|
||||
});
|
||||
|
||||
it("does not tag opaque Markdown contexts", () => {
|
||||
const html = renderMarkdown("`#code` and $#math$ and <https://example.com/#link> and #ok");
|
||||
|
||||
expect(html).not.toContain('data-tag="code"');
|
||||
expect(html).not.toContain('data-tag="math"');
|
||||
expect(html).not.toContain('data-tag="link"');
|
||||
expect(html).toContain('data-tag="ok"');
|
||||
});
|
||||
|
||||
it("keeps math opaque while the math renderer is loading", () => {
|
||||
const html = renderMarkdownWithoutMath("$#inline$ and $$#mismatch$ plus #ok\n\n$$\n#block\n$$");
|
||||
|
||||
expect(html).not.toContain('data-tag="inline"');
|
||||
expect(html).toContain('data-tag="mismatch"');
|
||||
expect(html).not.toContain('data-tag="block"');
|
||||
expect(html).toContain('data-tag="ok"');
|
||||
});
|
||||
|
||||
it("maps CRLF flow math while the math renderer is loading", () => {
|
||||
const html = renderMarkdownWithoutMath("$$\r\n#inside\r\n$$\r\n#outside");
|
||||
|
||||
expect(html).not.toContain('data-tag="inside"');
|
||||
expect(html).toContain('data-tag="outside"');
|
||||
});
|
||||
|
||||
it.each([
|
||||
["$#inner\\$#outer", "$#inner$"],
|
||||
["$#inner&$#outer", "$#inner&$"],
|
||||
])("maps decoded math source before an outside tag: %s", (source, renderedPrefix) => {
|
||||
const html = renderMarkdownWithoutMath(source);
|
||||
|
||||
expect(html).toContain(`${renderedPrefix}<span class="tag" data-tag="outer">#outer</span>`);
|
||||
expect(html).not.toContain('data-tag="inner"');
|
||||
});
|
||||
|
||||
it("maps CommonMark NUL replacement inside fallback math", () => {
|
||||
const html = renderMarkdownWithoutMath("$#inside\0$#outside");
|
||||
|
||||
expect(html).not.toContain('data-tag="inside"');
|
||||
expect(html).toContain('data-tag="outside"');
|
||||
});
|
||||
|
||||
it("keeps undeclared footnote definitions opaque", () => {
|
||||
const html = renderMarkdown("A[^1]\n\n[^1]: #foot\n\n#ok");
|
||||
|
||||
expect(html).not.toContain('data-tag="foot"');
|
||||
expect(html).toContain('data-tag="ok"');
|
||||
});
|
||||
|
||||
it("follows flow-math metadata boundaries", () => {
|
||||
expect(renderMarkdown("$$meta\n#math")).not.toContain('data-tag="math"');
|
||||
expect(renderMarkdown("$$meta$x\n#tag")).toContain('data-tag="tag"');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
53
web/tests/tag-grammar.test.ts
Normal file
53
web/tests/tag-grammar.test.ts
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { findTagMatches } from "@/utils/tag-grammar";
|
||||
|
||||
const values = (source: string) => findTagMatches(source).map((match) => match.value);
|
||||
|
||||
describe("tag scanner", () => {
|
||||
it.each([
|
||||
["#tag", ["tag"]],
|
||||
["hello#tag", ["tag"]],
|
||||
["#标签", ["标签"]],
|
||||
["#2026", ["2026"]],
|
||||
["#C++", ["C++"]],
|
||||
["#R&D", ["R&D"]],
|
||||
["#-foo #foo- #--- #&&", ["-foo", "foo-", "---", "&&"]],
|
||||
["#work/notes", ["work/notes"]],
|
||||
["#book/", ["book"]],
|
||||
["#/book", []],
|
||||
["#book//fiction", ["book"]],
|
||||
["#book/fiction/", ["book/fiction"]],
|
||||
["#l·l #foo‿bar", ["l·l", "foo‿bar"]],
|
||||
["#foo,bar #price€ #€budget #v²", ["foo", "price", "v"]],
|
||||
["#first#second", ["first", "second"]],
|
||||
["##tag", ["tag"]],
|
||||
["#tag ﹟tag", []],
|
||||
])("scans %s", (source, expected) => {
|
||||
expect(values(source)).toEqual(expected);
|
||||
});
|
||||
|
||||
it("emits ignored source code points only in the source span", () => {
|
||||
const source = "#AB #foo #AB #foo #A️B #́foo #café #foo/́bar";
|
||||
expect(values(source)).toEqual(["AB", "foo", "AB", "foo", "AB", "foo", "café", "foo/bar"]);
|
||||
expect(findTagMatches("#AB")[0]).toMatchObject({ from: 0, to: 4, source: "AB", value: "AB" });
|
||||
expect(values("# #́")).toEqual([]);
|
||||
});
|
||||
|
||||
it("matches fully-qualified emoji atomically and excludes components", () => {
|
||||
expect(values("#*️⃣ #‼️ #♥ #♥️ #🏻")).toEqual(["*️⃣", "‼️", "♥️"]);
|
||||
expect(values("#️⃣ ##️⃣ #first#️⃣")).toEqual(["#️⃣", "first#️⃣"]);
|
||||
});
|
||||
|
||||
it("uses the checked-in Unicode 17 and Emoji 17 repertoire", () => {
|
||||
expect(values("# #")).toEqual(["", ""]);
|
||||
});
|
||||
|
||||
it("has no tag-specific length limit", () => {
|
||||
const tag = "a".repeat(101);
|
||||
expect(values(`#${tag}`)).toEqual([tag]);
|
||||
});
|
||||
|
||||
it("does not match an emoji across the requested source limit", () => {
|
||||
expect(findTagMatches("#😀", 0, 2)).toEqual([]);
|
||||
});
|
||||
});
|
||||
17
web/tests/tag-unicode-parity.test.ts
Normal file
17
web/tests/tag-unicode-parity.test.ts
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
import { readFile } from "node:fs/promises";
|
||||
import { resolve } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
const DIGEST_PATTERN = /Unicode data SHA-256: ([0-9a-f]{64})/;
|
||||
|
||||
describe("tag Unicode data", () => {
|
||||
it("keeps the Go and TypeScript generated tables in sync", async () => {
|
||||
const [goSource, typeScriptSource] = await Promise.all([
|
||||
readFile(resolve(process.cwd(), "../internal/markdown/parser/tag_unicode_tables.go"), "utf8"),
|
||||
readFile(resolve(process.cwd(), "src/utils/tag-unicode-data.ts"), "utf8"),
|
||||
]);
|
||||
|
||||
expect(goSource.match(DIGEST_PATTERN)?.[1]).toBeDefined();
|
||||
expect(goSource.match(DIGEST_PATTERN)?.[1]).toBe(typeScriptSource.match(DIGEST_PATTERN)?.[1]);
|
||||
});
|
||||
});
|
||||
64
web/tests/tag.test.ts
Normal file
64
web/tests/tag.test.ts
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
import { create, fromBinary, toBinary } from "@bufbuild/protobuf";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { findTagMetadata, mergeTagCounts } from "@/lib/tag";
|
||||
import {
|
||||
type UserSetting_TagMetadata,
|
||||
type UserSetting_TagsSetting,
|
||||
UserSetting_TagsSettingSchema,
|
||||
UserStatsSchema,
|
||||
} from "@/types/proto/api/v1/user_service_pb";
|
||||
|
||||
describe("exact tag keys", () => {
|
||||
it("aggregates names that collide with Object prototype properties", () => {
|
||||
const source = Object.fromEntries([
|
||||
["toString", 1],
|
||||
["constructor", 2],
|
||||
["__proto__", 3],
|
||||
]);
|
||||
const counts = mergeTagCounts(source, { toString: 4 });
|
||||
|
||||
expect(Object.getPrototypeOf(counts)).toBeNull();
|
||||
expect(counts.toString).toBe(5);
|
||||
expect(counts.constructor).toBe(2);
|
||||
expect(counts.__proto__).toBe(3);
|
||||
});
|
||||
|
||||
it("preserves exact prototype-shaped tag keys through protobuf maps", () => {
|
||||
const source = Object.fromEntries([
|
||||
["normal", 1],
|
||||
["constructor", 2],
|
||||
["__proto__", 3],
|
||||
]);
|
||||
const encoded = toBinary(UserStatsSchema, create(UserStatsSchema, { tagCount: source }));
|
||||
const decoded = fromBinary(UserStatsSchema, encoded);
|
||||
const counts = mergeTagCounts(decoded.tagCount);
|
||||
|
||||
expect(Object.keys(decoded.tagCount).sort()).toEqual(["__proto__", "constructor", "normal"]);
|
||||
expect(counts.normal).toBe(1);
|
||||
expect(counts.constructor).toBe(2);
|
||||
expect(counts.__proto__).toBe(3);
|
||||
});
|
||||
|
||||
it("preserves exact prototype-shaped metadata keys when creating tag settings", () => {
|
||||
const source = Object.fromEntries([
|
||||
["normal", { blurContent: false }],
|
||||
["constructor", { blurContent: true }],
|
||||
["__proto__", { blurContent: true }],
|
||||
]);
|
||||
const created = create(UserSetting_TagsSettingSchema, { tags: source });
|
||||
const decoded = fromBinary(UserSetting_TagsSettingSchema, toBinary(UserSetting_TagsSettingSchema, created));
|
||||
|
||||
expect(Object.keys(created.tags).sort()).toEqual(["__proto__", "constructor", "normal"]);
|
||||
expect(Object.keys(decoded.tags).sort()).toEqual(["__proto__", "constructor", "normal"]);
|
||||
expect(decoded.tags.normal.blurContent).toBe(false);
|
||||
expect(decoded.tags.constructor.blurContent).toBe(true);
|
||||
expect(decoded.tags.__proto__.blurContent).toBe(true);
|
||||
});
|
||||
|
||||
it("does not treat inherited properties as exact metadata", () => {
|
||||
const metadata = { blurContent: true } as UserSetting_TagMetadata;
|
||||
const setting = { tags: { ".*": metadata } } as UserSetting_TagsSetting;
|
||||
|
||||
expect(findTagMetadata("toString", setting)).toBe(metadata);
|
||||
});
|
||||
});
|
||||
Loading…
Reference in a new issue