oxedyne/fe2o3/fe2o3_text/src/unicode/mod.rs
1.6 KiB, 7 runs
created by r1870400018:13904, which is this file's identity for as long as the history lasts, whatever it is later renamed to
download · who wrote it · its history
| 1 | //! Unicode text algorithms, implemented over tables generated from the Unicode Character Database. |
| 2 | //! |
| 3 | //! The crate carries an implementation of four of the Unicode annexes, each of which a document |
| 4 | //! renderer needs: |
| 5 | //! |
| 6 | //! - [`norm`], the normalisation forms of UAX #15. NFC and NFD, and their compatibility variants |
| 7 | //! NFKC and NFKD. |
| 8 | //! - [`linebreak`], the line breaking algorithm of UAX #14, which yields the byte offsets at which |
| 9 | //! a line may or must be broken. |
| 10 | //! - [`segment`], the grapheme cluster and word boundaries of UAX #29, which give a cursor |
| 11 | //! somewhere to land. |
| 12 | //! - [`bidi`], the bidirectional algorithm of UAX #9, which resolves embedding levels and the |
| 13 | //! visual order of a paragraph. |
| 14 | //! |
| 15 | //! Beside them, [`property`] answers which General_Category, Script, Script_Extensions and binary |
| 16 | //! properties a character has, and resolves the `\p{...}` names a regular expression uses. |
| 17 | //! |
| 18 | //! The tables in [`tables`] are generated and committed, never fetched at build or run time. The |
| 19 | //! Unicode version they come from is [`UCD_VERSION`]; the generator is |
| 20 | //! `fe2o3_text/src/bin/gen_unicode.rs`, and it also vendors the Unicode Consortium conformance |
| 21 | //! files that `tests/unicode.rs` runs against. |
| 22 | //! |
| 23 | //! ``` |
| 24 | //! use oxedyne_fe2o3_text::unicode::norm; |
| 25 | //! |
| 26 | //! assert_eq!(norm::nfc("e\u{0301}"), "é"); |
| 27 | //! assert_eq!(norm::nfd("é"), "e\u{0301}"); |
| 28 | //! ``` |
| 29 | //! |
| 30 | //! [Written with AI entirely](https://need2know.ai/entirely-ai/code)\ |
| 31 | //! Anthropic Claude |
| 32 | |
| 33 | pub mod bidi; |
| 34 | pub mod linebreak; |
| 35 | pub mod lookup; |
| 36 | pub mod norm; |
| 37 | pub mod property; |
| 38 | pub mod segment; |
| 39 | pub mod tables; |
| 40 | |
| 41 | pub use tables::prop; |
| 42 | pub use tables::UCD_VERSION; |