8.6 KiB, 58 runs
created by r1870400018:13, which is this file's identity for as long as the history lasts, whatever it is later renamed to
download · who wrote it · its history
| 1 | <p align="center"> |
| 2 | <img src="img/oxedyne_logo_white_text_below_opt.svg" width="150"/> |
| 3 | </p> |
| 4 | |
| 5 | <p align="center"> |
| 6 | <img src="img/fe2o3_display_white_opt.svg" width="200"/> |
| 7 | </p> |
| 8 | |
| 9 | # Oxedyne Hematite (fe2o3) |
| 10 | |
| 11 | Hematite is a collection of Rust crates that grew from an exploration into database design and key data structures. The project includes several innovative components: |
| 12 | |
| 13 | - **O3db**: A log-structured key-value database inspired by BitCask, designed for high throughput. |
| 14 | - **JDAT**: A superset of JSON adding types, combining human-readable and binary formats. |
| 15 | - **Shield**: A secure peer-to-peer protocol and app built on UDP. |
| 16 | - **Namex**: A distributed universal name codex for schemes and specifications. |
| 17 | |
| 18 | Oxedyne is my Rust/Web development shop. |
| 19 | |
| 20 | ## Status |
| 21 | |
| 22 | This solo project has been in progress on a part time basis for several years, and I estimate completion of the initial 0.9.0 release is at around 50% overall. I'm now doing this in public in the hope of soliciting ideas and suggestions, with a view to publishing a developer guide and accepting contributions some time in the next 12 months. |
| 23 | |
| 24 | Feel free to: |
| 25 | - Star/watch the repository if interested, |
| 26 | - Open issues for bugs or feature suggestions, |
| 27 | - Fork and experiment with the code, |
| 28 | - Provide feedback through discussions, |
| 29 | - Contribute through GitHub Sponsors if you want faster development or wider adoption, and |
| 30 | - Make pull requests once the contribution window opens. |
| 31 | |
| 32 | See the detailed [project progress](PROGRESS.md) for recently completed and upcoming features. |
| 33 | |
| 34 | ## Project Philosophy |
| 35 | |
| 36 | Hematite started as a Rust learning experience, and one of my objectives has been to minimise third-party dependencies. However, despite every effort to reinvent lots of wheels, it still depends on many crates directly and indirectly. So I am very grateful to all their authors and contributors. |
| 37 | |
| 38 | The code aims mostly to be correct, readable, and maintainable rather than super-fast or clever. While I have always strived for economy and efficiency, I'm looking forward to community contributions that help make better use of Rust features, and lead to more aggressive optimisation. |
| 39 | |
| 40 | One of the initial motivations for the library was to improve my personal developer experience, which has involved a range of experiments particularly around error handling and logging. The code avoids the use of `unsafe` and `unwrap`. The `?` propogator is avoided in favour of macros that are more explicitly function-like such as the directly replacement `ok!`, the match-based `res!` and the closure `catch!` which tries to capture unwind panics. Eventually any impacts on performance will be properly assessed. Tags were built into the `fe2o3_core::error::Error` type which offers a foundation for theoretical benefits but the practical value remains to be seen. |
| 41 | |
| 42 | ## Components |
| 43 | |
| 44 | The crates can be organised by their level of internal cross-dependencies, currently the most notable being: |
| 45 | |
| 46 | ### Foundational (no internal cross-dependencies) |
| 47 | - **fe2o3_core**: Core traits, utilities, logging, and fundamental types, typically small and relatively simple. |
| 48 | - **fe2o3_stds**: A place for existing public standards. |
| 49 | |
| 50 | ### Fundamental (1-3 internal cross-dependencies) |
| 51 | - **fe2o3_bot**: Thread worker library, for bot-like worker threads with common termination semantics. |
| 52 | - **fe2o3_data**: Specialised data structures such as ring buffers, stacks, trees and timestamped types. |
| 53 | - **fe2o3_iop_hash**: Interoperability layer for hashing, including SHA3-256 and Seahash. |
| 54 | - **fe2o3_net**: Network utilities with support for DNS, HTTP, WebSocket, and SMTP. |
| 55 | - **fe2o3_num**: Numerical type utilities, big integer, decimal types, and number strings. |
| 56 | - **fe2o3_text**: String manipulation and formatting, with text processing tools, base2x encoding and a REPL-friendly Stringer. |
| 57 | |
| 58 | ### Functional (4-7 internal cross-dependencies) |
| 59 | - **fe2o3_crypto**: Cryptography library including post-quantum implementations such as SABER and Dilithium. |
| 60 | - **fe2o3_hash**: Generic hashing and key derivation utilities. |
| 61 | - **fe2o3_iop_crypto**: Interoperability layer for cryptography, including AES-GCM, and Ed25519. |
| 62 | - **fe2o3_iop_db**: Interoperability layer for databases. |
| 63 | - **fe2o3_jdat**: A user-level type layer including JDAT format implementation. |
| 64 | - **fe2o3_namex**: A distributed, general purpose, universal name codex with utilities. |
| 65 | - **fe2o3_syntax**: A command parsing library for defining custom protocols. |
| 66 | |
| 67 | ### Application Level (8+ internal cross-dependencies) |
| 68 | - **fe2o3_o3db**: The Ozone database. |
| 69 | - **fe2o3_shield**: Shield protocol implementation for secure peer-to-peer networking. |
| 70 | - **fe2o3_steel**: A web server implementation with developer mode, HTTPS, WebSocket, and SMTPS support. |
| 71 | - **fe2o3_tui**: A terminal user interface library, including the Ironic TUI. |
| 72 | |
| 73 | ## Getting Started |
| 74 | |
| 75 | Hematite is currently available via [GitHub]("https://github.com/oxedyne-io/fe2o3.git") and version 0.9.0 will be available on [crates.io](crates.io). You can use it in several ways: |
| 76 | |
| 77 | ### Installation |
| 78 | |
| 79 | Note that Hematite is developed on linux. Compilation of C static libraries for the post quantum cryptography scheme SABRE reference implementation (for testing) requires OpenSSL. On ubuntu: |
| 80 | |
| 81 | sudo apt-get install libssl-dev |
| 82 | |
| 83 | ### From GitHub |
| 84 | |
| 85 | For the latest development version, use git dependencies: |
| 86 | ```toml |
| 87 | [dependencies] |
| 88 | oxedyne_fe2o3 = { git = "https://github.com/oxedyne-io/fe2o3" } |
| 89 | ``` |
| 90 | |
| 91 | ### Local Development |
| 92 | |
| 93 | To explore or contribute: |
| 94 | |
| 95 | 1. Clone the repository: |
| 96 | ```bash |
| 97 | git clone https://github.com/oxedyne-io/fe2o3.git |
| 98 | cd fe2o3 |
| 99 | ``` |
| 100 | |
| 101 | 2. Build the project: |
| 102 | ```bash |
| 103 | cargo build |
| 104 | ``` |
| 105 | |
| 106 | 3. Run the tests: |
| 107 | ```bash |
| 108 | cargo test |
| 109 | ``` |
| 110 | |
| 111 | Tests are grouped inside files within a `tests` directory for each crate, and run via the `tests/main.rs` file, e.g.: |
| 112 | ```bash |
| 113 | cd fe2o3_core |
| 114 | cargo test main -- --nocapture |
| 115 | ``` |
| 116 | |
| 117 | ### From crates.io |
| 118 | |
| 119 | When it becomes available, you can access specific crates directly : |
| 120 | ```toml |
| 121 | [dependencies] |
| 122 | oxedyne_fe2o3_core = "0.5.0" |
| 123 | oxedyne_fe2o3_jdat = "0.5.0" |
| 124 | ``` |
| 125 | |
| 126 | Or use the workspace crate to access everything: |
| 127 | ```toml |
| 128 | [dependencies] |
| 129 | oxedyne_fe2o3 = { version = "0.5.0", features = ["all"] } |
| 130 | ``` |
| 131 | |
| 132 | Or just the components you need: |
| 133 | ```toml |
| 134 | [dependencies] |
| 135 | oxedyne_fe2o3 = { version = "0.5.0", features = ["core", "jdat", "o3db"] } |
| 136 | ``` |
| 137 | |
| 138 | ### Usage Examples |
| 139 | |
| 140 | Check the test files in each crate's `tests` directory for detailed usage examples. Here's a quick start: |
| 141 | |
| 142 | ```rust |
| 143 | // Using the workspace crate |
| 144 | use oxedyne_fe2o3::core::prelude::*; |
| 145 | use oxedyne_fe2o3::jdat; |
| 146 | |
| 147 | // Or individual crates |
| 148 | use oxedyne_fe2o3_core::prelude::*; |
| 149 | use oxedyne_fe2o3_jdat; |
| 150 | ``` |
| 151 | |
| 152 | ## Highlights |
| 153 | |
| 154 | ### Ozone Database |
| 155 | |
| 156 | O3db aims to: |
| 157 | - Write data quickly to log-structured files, |
| 158 | - Offer a high degree of parallalism using operating system threads, |
| 159 | - Keep as much data in a fast volatile cache as possible, |
| 160 | - Provide automatic file garbage collection, and a |
| 161 | - Simple, familiar key-value interface. |
| 162 | |
| 163 | ### JDAT |
| 164 | |
| 165 | JDAT extends JSON to support: |
| 166 | - Type annotations including multiple number formats, |
| 167 | - Binary encoding and decoding, |
| 168 | - Comment support, and |
| 169 | - Any type as map keys. |
| 170 | |
| 171 | ### Shield Protocol |
| 172 | |
| 173 | Shield stands for Signed Hash In Every Little Datagram and provides: |
| 174 | - UDP-based messaging, |
| 175 | - Secure and authenticated peer-to-peer communication, |
| 176 | - Post-quantum cryptography options, and |
| 177 | - Mitigates DOS attacks through use of proof of work in all packets. |
| 178 | |
| 179 | ### Namex |
| 180 | |
| 181 | Namex aims to enable: |
| 182 | - A decentralised, versionless database for mapping 32-byte identifiers to entities |
| 183 | - Support for over 4 billion identifier families, each with over 65,000 variants |
| 184 | - JDAT-based storage format with human-readable export options |
| 185 | - Rich entity metadata including names, descriptions, timestamps, and cross-references |
| 186 | - Language-aware naming with alternative name support |
| 187 | - Hierarchical tagging system for flexible categorisation |
| 188 | |
| 189 | ### Steel Web Server |
| 190 | |
| 191 | Steel includes: |
| 192 | - A robust HTTPS server with WebSocket upgrade support |
| 193 | - Development mode with hot reloading and self-signed certificates |
| 194 | - Production mode with Let's Encrypt certificate automation |
| 195 | - Built-in static file serving and route configuration |
| 196 | - Clean separation between server and application layers |
| 197 | - JavaScript/TypeScript bundling and SASS compilation in development mode |
| 198 | |
| 199 | ### Iron Interactive Console |
| 200 | |
| 201 | Ironic aims to offer: |
| 202 | - Modal editing inspired by Vi/Vim |
| 203 | - Windowing system with draggable/resizable windows |
| 204 | - Tab-based interface for organizing content |
| 205 | - Customisable styles and colours |
| 206 | - Scrollbars and status indicators |
| 207 | - File tree navigation |
| 208 | - Support for both static and editable text views |
| 209 | |
| 210 | ## License |
| 211 | |
| 212 | See the [LICENSE](LICENSE) file for license rights and limitations. |
| 213 | |
| 214 | ## Future Plans |
| 215 | |
| 216 | - Complete initial implementation of all components, |
| 217 | - Stabilise APIs, |
| 218 | - Expand documentation, |
| 219 | - Open for community contributions, |
| 220 | - Publish to crates.io. |
| 221 | |
| 222 | ## Contact |
| 223 | |
| 224 | <hello@oxedyne.io> |