Oregami
Repositories/oxedyne/fe2o3

oxedyne/fe2o3/fe2o3_steel/src/srv/publish/outbox.rs

56.9 KiB, 91 runs

created by r1870400018:68852, which is this file's identity for as long as the history lasts, whatever it is later renamed to

download · who wrote it · its history

1//! The paced outbound queue: what a site sends, held in its own store and let out at a set rate.
2//!
3//! A sign-up used to wait on SMTP inside the request, so the pace of a site's mail was whatever the
4//! visitors made it, and a script could spend the host's sending reputation as fast as it could post
5//! a form. Now the request writes a queue entry and answers. A drainer task takes entries off the
6//! queue and sends them at no more than the host's hourly ceiling, and a ceiling of 0 sends nothing
7//! at all, which is how a site is moved live with its mail tap shut.
8//!
9//! # Where the queue lives
10//!
11//! In the vhost's own Ozone store, one lane for each [`Kind`]. A lane is a head counter, a tail
12//! counter and an entry at each sequence number between them, so the queue survives a restart and
13//! holds its place. Head and tail move only inside [`store::exclusive`]. **No token is stored in an
14//! entry.** At send time the drainer reads the subscriber afresh and sends to nobody who has left
15//! the state the entry was queued for, so the link a message carries is always the current one.
16//!
17//! # Where the ceiling lives
18//!
19//! In one [`Pacer`] shared by every drainer on the host, because the address and the domain carry
20//! one reputation, not one for each site. The pacer spaces sends evenly (no bursts), and it is held
21//! in memory, so a restart can add at most one send to the schedule.
22//!
23//! # The two lanes, and the alert
24//!
25//! A confirmation waits in the `c` lane and the newsletter in the `n` lane. The drainer takes from `c`
26//! first, so a sign-up is not kept behind a newsletter, but a [`Share`] gives the newsletter the next
27//! slot after [`CONFIRM_RUN`] confirmations in a row, so a stream of sign-ups cannot starve it. The `c`
28//! lane is also capped in depth (see [`push_capped`]). A [`Watch`] reads the queue after each pass and
29//! tells the operator once when the oldest entry has waited past `outbox_alert_secs`, which a held
30//! ceiling and a deep queue both cause, and once more when the queue has drained.
31//!
32//! # A far end that does not answer
33//!
34//! The drainer is one task for a vhost, so a delivery that never finishes would stop everything behind
35//! it. Each delivery therefore runs under the [`Pacer`]'s deadline. One that overruns is a transient
36//! failure, and the pacer also blocks the mail exchange the delivery was waiting on, by its address, for
37//! a time that doubles whenever that exchange times out again. A name is no key to a far end that
38//! answers for any name: a wildcard MX hands every fresh subdomain the same exchange, so a block on the
39//! recipient's domain would cost one deadline for each. Before an entry is sent the drainer asks the
40//! [`Courier`] for the recipient's exchanges and passes the blocked ones in a [`Dial`], so the delivery
41//! goes to another exchange of the domain if there is one. An entry whose exchanges are all blocked is
42//! put back until the earliest block ends, using no slot, and an exchange is forgotten when a delivery
43//! to it is answered. The pacer holds at most [`SLOW_MAX`] blocks, so a flood of names pointing at
44//! different hosts cannot grow it without bound.
45//!
46//! # The daily sweep
47//!
48//! The same task clears, once a day, what has outlived its use: an unconfirmed sign-up whose last
49//! confirmation was sent `pending_expiry_days` ago, and the rate rows whose windows have passed. See
50//! [`sweep`].
51//!
52//! [Written with AI entirely](https://need2know.ai/entirely-ai/code)\
53//! Anthropic Claude
54
55use crate::srv::{
56 alert::{
57 AlertEvent,
58 Alerter,
59 },
60 publish::{
61 PostState,
62 PublishConfig,
63 comment,
64 rate::{
65 self,
66 Window,
67 },
68 send::{
69 self,
70 Fate,
71 },
72 store::{
73 self,
74 Edit,
75 },
76 subscribe::{
77 self,
78 SubState,
79 },
80 },
81};
82
83use oxedyne_fe2o3_core::prelude::*;
84use oxedyne_fe2o3_iop_crypto::enc::Encrypter;
85use oxedyne_fe2o3_iop_db::api::Database;
86use oxedyne_fe2o3_iop_hash::api::Hasher;
87use oxedyne_fe2o3_jdat::{
88 prelude::*,
89 id::NumIdDat,
90};
91use oxedyne_fe2o3_mail::outbound::retry_after;
92use oxedyne_fe2o3_net::smtp::client::{
93 Dial,
94 is_permanent,
95 maybe_sent,
96};
97
98use std::{
99 collections::HashMap,
100 future::Future,
101 net::IpAddr,
102 sync::{
103 Arc,
104 Mutex,
105 RwLock,
106 },
107 time::{
108 Duration,
109 Instant,
110 },
111};
112
113use tokio::sync::Notify;
114
115
116pub const PREFIX: &str = "publish/outbox/";
117
118// The ceiling a mail block takes where it names none. Chosen to be slow enough that a young
119// domain is never mistaken for a spammer, and fast enough that one confirmation is never kept
120// waiting more than a minute.
121pub const HOURLY_DEFAULT: u32 = 100;
122
123// How many times an entry is tried before it is given up and dropped.
124pub const TRIES_MAX: u32 = 5;
125
126// The longest the drainer sleeps with nothing to wake it, in seconds. The queue is also polled at
127// this interval, so an entry written by another process is found within it.
128pub const POLL_SECS: u64 = 30;
129
130// Entries written under one write guard. A newsletter to thousands is pushed in chunks so that
131// reads on the same store are not kept waiting for the whole run.
132const PUSH_CHUNK: usize = 64;
133
134const HOUR_MS: u64 = 3_600_000;
135
136// The lanes, in the order they are drained.
137const KINDS: [Kind; 2] = [Kind::Confirm, Kind::News];
138
139const DAY_SECS: u64 = 86_400;
140
141// How many places from the head of a lane to look for an entry that can be read, when the head
142// itself cannot be. A queue is not walked from end to end to find its age.
143const HEAD_LOOK: u64 = 16;
144
145// How long one delivery may take, in seconds, before the drainer gives up on it and goes on. The
146// SMTP client allows a minute for each step and ten for the end of the message, which a far end
147// that answers slowly can spend in full on every exchange address it has.
148pub const DEADLINE_SECS: u64 = 150;
149
150// How long a mail exchange that has timed out is left alone, in seconds: the first time, and the most it
151// grows to by doubling each time it times out again.
152pub const BACKOFF_MIN_SECS: u64 = 600;
153pub const BACKOFF_MAX_SECS: u64 = 6 * 3600;
154
155// The most exchanges the pacer remembers as timed out. At the bound the block that ends soonest makes
156// room for a new one.
157pub const SLOW_MAX: usize = 64;
158
159// How many confirmations may take the slots one after another before the newsletter is next.
160pub const CONFIRM_RUN: u32 = 3;
161
162// The most entries one call of `take` looks at. The write guard is held for a call, so a long lane of
163// entries backing off is looked at in parts, with the guard let go between them.
164pub const TAKE_LOOK: u64 = 256;
165
166
167// ┌───────────────────────────────────────────────────────────────────────────┐
168// │ PACER │
169// └───────────────────────────────────────────────────────────────────────────┘
170
171/// What the pacer says about sending now.
172#[derive(Clone, Copy, Debug, Eq, PartialEq)]
173pub enum Pace {
174 Held, // the ceiling is 0: nothing leaves
175 Wait(u64), // the next slot is this many milliseconds away
176 Go, // this send holds the slot
177}
178
179/// The host's outbound ceiling, shared by every drainer.
180///
181/// Sends are spaced evenly: the gap is an hour divided by the ceiling, rounded up, so the ceiling
182/// is never exceeded in any hour. The next slot is held in memory only.
183#[derive(Debug)]
184pub struct Pacer {
185 hourly: u32, // sends an hour; 0 holds everything
186 deadline: Duration, // the longest one delivery may take
187 next: Mutex<u64>, // the earliest millisecond the next send may leave
188 slow: Mutex<HashMap<IpAddr, Backoff>>, // mail exchanges that have timed out
189 wake: Notify, // signalled when something is queued
190}
191
192// How long an exchange is left alone, and the wait that set it, for the next doubling.
193#[derive(Clone, Copy, Debug)]
194struct Backoff {
195 until: u64, // unix second the block ends
196 wait: u64, // seconds the block was set for
197}
198
199impl Pacer {
200
201 pub fn new(hourly: u32) -> Self {
202 Self {
203 hourly,
204 deadline: Duration::from_secs(DEADLINE_SECS),
205 next: Mutex::new(0),
206 slow: Mutex::new(HashMap::new()),
207 wake: Notify::new(),
208 }
209 }
210
211 /// This pacer with `deadline` as the longest one delivery may take, where it would otherwise be
212 /// [`DEADLINE_SECS`].
213 pub fn with_deadline(mut self, deadline: Duration) -> Self {
214 self.deadline = deadline;
215 self
216 }
217
218 pub fn deadline(&self) -> Duration {
219 self.deadline
220 }
221
222 pub fn hourly(&self) -> u32 {
223 self.hourly
224 }
225
226 /// Is the ceiling 0, so that nothing leaves?
227 pub fn is_held(&self) -> bool {
228 self.hourly == 0
229 }
230
231 /// The milliseconds between two sends at this ceiling; 0 where the ceiling is 0.
232 pub fn gap_ms(&self) -> u64 {
233 if self.hourly == 0 {
234 return 0;
235 }
236 let h = self.hourly as u64;
237 (HOUR_MS + h - 1) / h
238 }
239
240 /// Asks for the slot at `now_ms`. A `Go` has taken it: whatever the send does, the next one
241 /// waits a gap. The slot is taken before the send, so a send that fails still counts.
242 pub fn claim(&self, now_ms: u64) -> Outcome<Pace> {
243 if self.hourly == 0 {
244 return Ok(Pace::Held);
245 }
246 let mut next = lock_mutex!(self.next);
247 if now_ms < *next {
248 return Ok(Pace::Wait(*next - now_ms));
249 }
250 *next = now_ms.saturating_add(self.gap_ms());
251 Ok(Pace::Go)
252 }
253
254 /// Wakes every drainer that is idle, after something has been queued.
255 pub fn wake(&self) {
256 self.wake.notify_waiters();
257 }
258
259 /// Is the slot free at `now_ms`? Answers as [`Self::claim`] would, and takes nothing.
260 pub fn ready(&self, now_ms: u64) -> Outcome<Pace> {
261 if self.hourly == 0 {
262 return Ok(Pace::Held);
263 }
264 let next = lock_mutex!(self.next);
265 if now_ms < *next {
266 return Ok(Pace::Wait(*next - now_ms));
267 }
268 Ok(Pace::Go)
269 }
270
271 /// Until when, in unix seconds, is a mail exchange left alone for having timed out? `None` where it
272 /// is not.
273 pub fn blocked(&self, ip: IpAddr, now: u64) -> Outcome<Option<u64>> {
274 let slow = lock_mutex!(self.slow);
275 Ok(slow.get(&ip).filter(|b| b.until > now).map(|b| b.until))
276 }
277
278 /// Which of `ips` are left alone at `now`, and until when each, in unix seconds.
279 pub fn blocks(&self, ips: &[IpAddr], now: u64) -> Outcome<Vec<(IpAddr, u64)>> {
280 let slow = lock_mutex!(self.slow);
281 Ok(ips.iter()
282 .filter_map(|ip| slow.get(ip).filter(|b| b.until > now).map(|b| (*ip, b.until)))
283 .collect())
284 }
285
286 /// Notes that a delivery waiting on the exchange at `ip` overran the deadline, and answers when the
287 /// exchange may next be tried: [`BACKOFF_MIN_SECS`] from now, then twice the last wait each time it
288 /// times out again, to [`BACKOFF_MAX_SECS`]. An exchange is remembered for a day after its block
289 /// ends, so a host that times out again as soon as it is tried is not started afresh, and then
290 /// forgotten. No more than [`SLOW_MAX`] are kept: a new one at the bound takes the place of the one
291 /// whose block ends soonest.
292 pub fn timed_out(&self, ip: IpAddr, now: u64) -> Outcome<u64> {
293 let mut slow = lock_mutex!(self.slow);
294 slow.retain(|_, b| now < b.until.saturating_add(DAY_SECS));
295 let wait = match slow.get(&ip) {
296 Some(b) => b.wait.saturating_mul(2).min(BACKOFF_MAX_SECS),
297 None => BACKOFF_MIN_SECS,
298 };
299 let until = now.saturating_add(wait);
300 if slow.len() >= SLOW_MAX && !slow.contains_key(&ip) {
301 let soonest = slow.iter().min_by_key(|(_, b)| b.until).map(|(k, _)| *k);
302 if let Some(k) = soonest {
303 slow.remove(&k);
304 }
305 }
306 slow.insert(ip, Backoff { until, wait });
307 Ok(until)
308 }
309
310 /// Forgets an exchange's block, since a delivery to it was answered.
311 pub fn reached(&self, ip: IpAddr) -> Outcome<()> {
312 let mut slow = lock_mutex!(self.slow);
313 slow.remove(&ip);
314 Ok(())
315 }
316}
317
318/// How the drainer divides its slots between the lanes: confirmations first, but the newsletter next
319/// once [`CONFIRM_RUN`] confirmations have taken slots in a row. One for each vhost's drainer.
320#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
321pub struct Share {
322 run: u32, // confirmation slots taken since a newsletter slot
323}
324
325impl Share {
326
327 // The lanes in the order the next pass looks at them.
328 fn order(&self) -> [Kind; 2] {
329 if self.run >= CONFIRM_RUN {
330 [Kind::News, Kind::Confirm]
331 } else {
332 [Kind::Confirm, Kind::News]
333 }
334 }
335
336 // Notes that a send of `kind` has taken a slot.
337 fn took(&mut self, kind: Kind) {
338 match kind {
339 Kind::Confirm => self.run = self.run.saturating_add(1),
340 Kind::News => self.run = 0,
341 }
342 }
343}
344
345/// The unix millisecond now, or 0 where the clock is before the epoch.
346pub fn now_ms() -> u64 {
347 std::time::SystemTime::now()
348 .duration_since(std::time::UNIX_EPOCH)
349 .map(|d| d.as_millis() as u64)
350 .unwrap_or(0)
351}
352
353
354// ┌───────────────────────────────────────────────────────────────────────────┐
355// │ COURIER │
356// └───────────────────────────────────────────────────────────────────────────┘
357
358/// What the drainer sends through: the host's signing sender in service, and a recorder in a test.
359pub trait Courier {
360
361 /// The From address a site that names none sends from.
362 fn default_from(&self) -> &str;
363
364 /// The addresses of the mail exchanges a message to `to` would be dialled at, in preference order,
365 /// found without dialling any. A courier that cannot tell answers none, and the drainer sends
366 /// without avoiding any exchange.
367 fn exchanges(&self, _to: &str) -> impl Future<Output = Outcome<Vec<IpAddr>>> + Send {
368 async { Ok(Vec::new()) }
369 }
370
371 /// Delivers one finished message to one recipient and answers the remote's queue id. `dial` names
372 /// the exchanges to keep away from, and the courier notes in it each exchange it dials, and marks it
373 /// whole once the message has gone (see [`Dial::whole`]), so the drainer can tell which one held a
374 /// delivery up that it gave up on, and whether that delivery may have been taken.
375 fn deliver(&self, from: &str, to: &str, msg: &str, dial: &Dial)
376 -> impl Future<Output = Outcome<String>> + Send;
377}
378
379
380// ┌───────────────────────────────────────────────────────────────────────────┐
381// │ ENTRY │
382// └───────────────────────────────────────────────────────────────────────────┘
383
384/// The lane an entry waits in.
385#[derive(Clone, Copy, Debug, Eq, PartialEq)]
386pub enum Kind {
387 Confirm, // a sign-up: applied, then its confirmation sent, drained first
388 News, // one subscriber's copy of a post
389}
390
391impl Kind {
392
393 pub fn as_str(&self) -> &'static str {
394 match self {
395 Self::Confirm => "confirm",
396 Self::News => "news",
397 }
398 }
399
400 pub fn of(s: &str) -> Option<Self> {
401 match s {
402 "confirm" => Some(Self::Confirm),
403 "news" => Some(Self::News),
404 _ => None,
405 }
406 }
407
408 fn lane(&self) -> &'static str {
409 match self {
410 Self::Confirm => "c",
411 Self::News => "n",
412 }
413 }
414}
415
416/// One queued send. It names who and what, never a token.
417#[derive(Clone, Debug, Eq, PartialEq)]
418pub struct Entry {
419 pub kind: Kind,
420 pub email: String, // normalised
421 pub slug: String, // the post, for a newsletter entry; empty otherwise
422 pub send: u64, // the place of the send in the history, for a newsletter entry; 0 otherwise
423 pub enqueued: u64, // unix seconds the entry was first queued, kept through retries
424 pub tries: u32, // attempts that have failed
425 pub next_try: u64, // unix seconds before which the entry is not tried
426 pub whole: bool, // its message went whole to a server that has not answered: maybe sent
427}
428
429impl Entry {
430
431 /// A fresh entry, due at once.
432 pub fn new(kind: Kind, email: &str, slug: &str, now: u64) -> Self {
433 Self {
434 kind,
435 email: email.to_string(),
436 slug: slug.to_string(),
437 send: 0,
438 enqueued: now,
439 tries: 0,
440 next_try: now,
441 whole: false,
442 }
443 }
444
445 /// This entry, counted on the history entry at `index`.
446 pub fn for_send(mut self, index: u64) -> Self {
447 self.send = index;
448 self
449 }
450
451 pub fn to_dat(&self) -> Dat {
452 let mut m = DaticleMap::new();
453 m.insert(dat!("kind"), dat!(self.kind.as_str().to_string()));
454 m.insert(dat!("email"), dat!(self.email.clone()));
455 if !self.slug.is_empty() {
456 m.insert(dat!("slug"), dat!(self.slug.clone()));
457 m.insert(dat!("send"), dat!(self.send));
458 }
459 m.insert(dat!("enqueued"), dat!(self.enqueued));
460 m.insert(dat!("tries"), dat!(self.tries as u64));
461 m.insert(dat!("next_try"), dat!(self.next_try));
462 if self.whole {
463 m.insert(dat!("whole"), dat!(true));
464 }
465 Dat::Map(m)
466 }
467
468 pub fn from_dat(d: &Dat) -> Outcome<Self> {
469 let m = match d {
470 Dat::Map(m) => m,
471 _ => return Err(err!(
472 "publish: an outbox entry must be a map, not {:?}.", d.kind();
473 Invalid, Input, Mismatch)),
474 };
475 let text = |key: &str| -> String {
476 match m.get(&dat!(key)) {
477 Some(Dat::Str(s)) => s.clone(),
478 _ => String::new(),
479 }
480 };
481 let kind = res!(Kind::of(&text("kind")).ok_or_else(|| err!(
482 "publish: an outbox entry names the kind '{}', which is not one.", text("kind");
483 Invalid, Input, Mismatch)));
484 let email = text("email");
485 if email.is_empty() {
486 return Err(err!("publish: an outbox entry names no email."; Invalid, Input, Missing));
487 }
488 Ok(Self {
489 kind,
490 email,
491 slug: text("slug"),
492 send: number(m.get(&dat!("send"))),
493 enqueued: number(m.get(&dat!("enqueued"))),
494 tries: number(m.get(&dat!("tries"))).min(u32::MAX as u64) as u32,
495 next_try: number(m.get(&dat!("next_try"))),
496 whole: matches!(m.get(&dat!("whole")), Some(Dat::Bool(true))),
497 })
498 }
499}
500
501// A stored count, however narrowly the decoder happened to type it. An absent or odd value is 0.
502fn number(d: Option<&Dat>) -> u64 {
503 match d {
504 Some(Dat::U64(x)) => *x,
505 Some(Dat::U32(x)) => *x as u64,
506 Some(Dat::U16(x)) => *x as u64,
507 Some(Dat::U8(x)) => *x as u64,
508 _ => 0,
509 }
510}
511
512
513// ┌───────────────────────────────────────────────────────────────────────────┐
514// │ QUEUE │
515// └───────────────────────────────────────────────────────────────────────────┘
516
517fn head_key(kind: Kind) -> Dat {
518 dat!(fmt!("{}{}/head", PREFIX, kind.lane()))
519}
520
521fn tail_key(kind: Kind) -> Dat {
522 dat!(fmt!("{}{}/tail", PREFIX, kind.lane()))
523}
524
525fn entry_key(kind: Kind, seq: u64) -> Dat {
526 dat!(fmt!("{}{}/{}", PREFIX, kind.lane(), seq))
527}
528
529// The counter at a key, on a database already locked. An absent one is 0.
530fn counter_in<
531 const UIDL: usize,
532 UID: NumIdDat<UIDL>,
533 ENC: Encrypter,
534 KH: Hasher,
535 DB: Database<UIDL, UID, ENC, KH>,
536>(
537 dbr: &DB,
538 key: &Dat,
539)
540 -> Outcome<u64>
541{
542 match res!(dbr.get(key, None)) {
543 Some((v, _)) => Ok(number(Some(&v))),
544 None => Ok(0),
545 }
546}
547
548// The entry at a sequence number, on a database already locked. A key never written, or deleted, is
549// `None`.
550fn entry_in<
551 const UIDL: usize,
552 UID: NumIdDat<UIDL>,
553 ENC: Encrypter,
554 KH: Hasher,
555 DB: Database<UIDL, UID, ENC, KH>,
556>(
557 dbr: &DB,
558 kind: Kind,
559 seq: u64,
560)
561 -> Outcome<Option<Entry>>
562{
563 match res!(dbr.get(&entry_key(kind, seq), None)) {
564 Some((v, _)) => Ok(Some(res!(Entry::from_dat(&v)))),
565 None => Ok(None),
566 }
567}
568
569// Writes an entry at the tail and moves the tail past it, on a database already write-locked. The
570// entry is written first, so a stop between the two leaves an entry nobody names and not a name
571// with no entry.
572fn push_in<
573 const UIDL: usize,
574 UID: NumIdDat<UIDL>,
575 ENC: Encrypter,
576 KH: Hasher,
577 DB: Database<UIDL, UID, ENC, KH>,
578>(
579 dbr: &DB,
580 user: UID,
581 e: &Entry,
582)
583 -> Outcome<()>
584{
585 store::edit_in(dbr, user, &tail_key(e.kind), |old| -> Outcome<(Edit, ())> {
586 let t = number(old.as_ref());
587 res!(dbr.insert(entry_key(e.kind, t), e.to_dat(), user, None));
588 Ok((Edit::Set(dat!(t + 1)), ()))
589 })
590}
591
592// Deletes the entry at `seq` and, where it is the head, moves the head past it.
593fn pop_in<
594 const UIDL: usize,
595 UID: NumIdDat<UIDL>,
596 ENC: Encrypter,
597 KH: Hasher,
598 DB: Database<UIDL, UID, ENC, KH>,
599>(
600 dbr: &DB,
601 user: UID,
602 kind: Kind,
603 seq: u64,
604)
605 -> Outcome<()>
606{
607 res!(dbr.delete(&entry_key(kind, seq), user, None));
608 store::edit_in(dbr, user, &head_key(kind), |old| -> Outcome<(Edit, ())> {
609 if number(old.as_ref()) == seq {
610 Ok((Edit::Set(dat!(seq + 1)), ()))
611 } else {
612 Ok((Edit::Keep, ()))
613 }
614 })
615}
616
617/// Is an entry still queued at `seq` in a lane, on a database already locked?
618///
619/// The drainer is handed an entry by [`take`] and acts on it later, so an erasure in between can have
620/// removed it. This is how an act that would make a record asks first, under the same guard as the
621/// act.
622pub fn queued_at_in<
623 const UIDL: usize,
624 UID: NumIdDat<UIDL>,
625 ENC: Encrypter,
626 KH: Hasher,
627 DB: Database<UIDL, UID, ENC, KH>,
628>(
629 dbr: &DB,
630 kind: Kind,
631 seq: u64,
632)
633 -> Outcome<bool>
634{
635 Ok(res!(entry_in(dbr, kind, seq)).is_some())
636}
637
638/// Removes every entry queued for an address, in both lanes, on a database already write-locked.
639/// Answers how many went.
640///
641/// For an erasure, which runs it under the guard that deletes the record, so that no entry survives
642/// to make the record again or to send to the address. A newsletter copy that goes is counted
643/// [`Fate::Left`] on its send, as one that finds its subscriber gone is. A history that will not take
644/// the count is logged and does not stop the erasure. The lane's head moves past the places this
645/// leaves empty at its front. `email` is normalised.
646pub fn purge_in<
647 const UIDL: usize,
648 UID: NumIdDat<UIDL>,
649 ENC: Encrypter,
650 KH: Hasher,
651 DB: Database<UIDL, UID, ENC, KH>,
652>(
653 dbr: &DB,
654 user: UID,
655 email: &str,
656)
657 -> Outcome<usize>
658{
659 let mut gone = 0;
660 for kind in KINDS {
661 let start = res!(counter_in(dbr, &head_key(kind)));
662 let tail = res!(counter_in(dbr, &tail_key(kind)));
663 let mut head = start;
664 let mut front = true; // every place so far is empty
665 for seq in start..tail {
666 match entry_in(dbr, kind, seq) {
667 Ok(Some(e)) if subscribe::normalise_email(&e.email) == email => {
668 res!(dbr.delete(&entry_key(kind, seq), user, None));
669 gone += 1;
670 if kind == Kind::News {
671 if let Err(e2) = send::tally_in(dbr, user, e.send as usize, Fate::Left) {
672 warn!("publish: could not count an erased newsletter copy as left: {}", e2);
673 }
674 }
675 }
676 // Left for `take`, which drops what it cannot read.
677 Ok(Some(_)) | Err(_) => front = false,
678 Ok(None) => {}
679 }
680 if front {
681 head = seq + 1;
682 }
683 }
684 if head != start {
685 res!(dbr.insert(head_key(kind), dat!(head), user, None));
686 }
687 }
688 Ok(gone)
689}
690
691/// Queues entries at the tail of their lanes, in order. Answers how many were queued.
692pub fn push<
693 const UIDL: usize,
694 UID: NumIdDat<UIDL>,
695 ENC: Encrypter,
696 KH: Hasher,
697 DB: Database<UIDL, UID, ENC, KH>,
698>(
699 db: &(Arc<RwLock<DB>>, UID),
700 entries: &[Entry],
701)
702 -> Outcome<usize>
703{
704 for chunk in entries.chunks(PUSH_CHUNK) {
705 res!(store::exclusive(db, |dbr, user| -> Outcome<()> {
706 for e in chunk {
707 res!(push_in(dbr, user, e));
708 }
709 Ok(())
710 }));
711 }
712 Ok(entries.len())
713}
714
715/// Queues one entry at the tail of its lane unless the lane already holds `max` entries (`0` is no
716/// limit). Answers whether it was queued.
717///
718/// The depth is read and the entry written under one guard, so no number of requests at once can
719/// carry a lane past its cap.
720pub fn push_capped<
721 const UIDL: usize,
722 UID: NumIdDat<UIDL>,
723 ENC: Encrypter,
724 KH: Hasher,
725 DB: Database<UIDL, UID, ENC, KH>,
726>(
727 db: &(Arc<RwLock<DB>>, UID),
728 entry: &Entry,
729 max: u64,
730)
731 -> Outcome<bool>
732{
733 store::exclusive(db, |dbr, user| -> Outcome<bool> {
734 if max > 0 && res!(queued_in(dbr, entry.kind)) >= max {
735 return Ok(false);
736 }
737 res!(push_in(dbr, user, entry));
738 Ok(true)
739 })
740}
741
742/// How many entries wait in a lane.
743pub fn queued<
744 const UIDL: usize,
745 UID: NumIdDat<UIDL>,
746 ENC: Encrypter,
747 KH: Hasher,
748 DB: Database<UIDL, UID, ENC, KH>,
749>(
750 db: &(Arc<RwLock<DB>>, UID),
751 kind: Kind,
752)
753 -> Outcome<u64>
754{
755 let (db_arc, _) = db;
756 let guard = lock_read!(db_arc);
757 queued_in(&*guard, kind)
758}
759
760/// As [`queued`], on a database already locked.
761pub fn queued_in<
762 const UIDL: usize,
763 UID: NumIdDat<UIDL>,
764 ENC: Encrypter,
765 KH: Hasher,
766 DB: Database<UIDL, UID, ENC, KH>,
767>(
768 dbr: &DB,
769 kind: Kind,
770)
771 -> Outcome<u64>
772{
773 let head = res!(counter_in(dbr, &head_key(kind)));
774 let tail = res!(counter_in(dbr, &tail_key(kind)));
775 Ok(tail.saturating_sub(head))
776}
777
778/// What a lane holds that can be sent now.
779#[derive(Clone, Debug, Eq, PartialEq)]
780pub enum Take {
781 Empty, // nothing queued
782 Later(u64), // everything is backing off; the soonest is due at this unix second
783 Due(u64, Entry), // the entry at this sequence number, still queued until [`done`]
784 More(Option<u64>, u64), // none due in the part looked at; the soonest so far, and how many are still to look at
785}
786
787/// Finds the first entry in a lane that is due at `now`, looking at no more than [`TAKE_LOOK`].
788///
789/// `left` is how many of the lane's entries are still to be looked at: `u64::MAX` to begin, then the
790/// number a [`Take::More`] answered. The caller makes the next call after letting writers in.
791///
792/// An entry backing off from a failure is moved to the tail, not waited for, so one address whose
793/// mail host is down never holds up the rest. Each is moved at most once in a call. A key that
794/// holds nothing, or nothing readable, is passed over, since a queue that stops at a bad entry
795/// would never move again.
796pub fn take<
797 const UIDL: usize,
798 UID: NumIdDat<UIDL>,
799 ENC: Encrypter,
800 KH: Hasher,
801 DB: Database<UIDL, UID, ENC, KH>,
802>(
803 db: &(Arc<RwLock<DB>>, UID),
804 kind: Kind,
805 now: u64,
806 left: u64,
807)
808 -> Outcome<Take>
809{
810 store::exclusive(db, |dbr, user| -> Outcome<Take> {
811 let mut head = res!(counter_in(dbr, &head_key(kind)));
812 let tail = res!(counter_in(dbr, &tail_key(kind)));
813 let left = tail.saturating_sub(head).min(left);
814 let mut look = left.min(TAKE_LOOK);
815 let rest = left - look;
816 let mut soonest: Option<u64> = None;
817 let mut out = Take::Empty;
818 let start = head;
819 while look > 0 {
820 look -= 1;
821 let e = match entry_in(dbr, kind, head) {
822 Ok(Some(e)) => e,
823 Ok(None) => {
824 head += 1;
825 continue;
826 }
827 Err(e) => {
828 warn!("publish: dropping an unreadable outbox entry {}/{}: {}",
829 kind.lane(), head, e);
830 res!(dbr.delete(&entry_key(kind, head), user, None));
831 head += 1;
832 continue;
833 }
834 };
835 if e.next_try <= now {
836 out = Take::Due(head, e);
837 break;
838 }
839 soonest = Some(match soonest {
840 Some(s) => s.min(e.next_try),
841 None => e.next_try,
842 });
843 res!(push_in(dbr, user, &e));
844 res!(dbr.delete(&entry_key(kind, head), user, None));
845 head += 1;
846 }
847 if head != start {
848 res!(dbr.insert(head_key(kind), dat!(head), user, None));
849 }
850 if let Take::Empty = out {
851 if rest > 0 {
852 out = Take::More(soonest, rest);
853 } else if let Some(s) = soonest {
854 out = Take::Later(s);
855 }
856 }
857 Ok(out)
858 })
859}
860
861/// Removes the entry [`take`] handed out, once it has been dealt with.
862pub fn done<
863 const UIDL: usize,
864 UID: NumIdDat<UIDL>,
865 ENC: Encrypter,
866 KH: Hasher,
867 DB: Database<UIDL, UID, ENC, KH>,
868>(
869 db: &(Arc<RwLock<DB>>, UID),
870 kind: Kind,
871 seq: u64,
872)
873 -> Outcome<()>
874{
875 store::exclusive(db, |dbr, user| pop_in(dbr, user, kind, seq))
876}
877
878/// As [`done`] for a newsletter message, counting how it ended on its send under the same guard.
879///
880/// The count is made only where the entry was still queued, so a message an erasure has already taken
881/// off the queue and counted is not counted twice. A history that will not take the count is logged,
882/// and the drainer carries on.
883fn finish<
884 const UIDL: usize,
885 UID: NumIdDat<UIDL>,
886 ENC: Encrypter,
887 KH: Hasher,
888 DB: Database<UIDL, UID, ENC, KH>,
889>(
890 db: &(Arc<RwLock<DB>>, UID),
891 entry: &Entry,
892 seq: u64,
893 fate: Fate,
894 id: &str,
895)
896 -> Outcome<()>
897{
898 store::exclusive(db, |dbr, user| -> Outcome<()> {
899 if !res!(queued_at_in(dbr, entry.kind, seq)) {
900 return Ok(());
901 }
902 res!(pop_in(dbr, user, entry.kind, seq));
903 if let Err(e) = send::tally_in(dbr, user, entry.send as usize, fate) {
904 warn!("{}: publish: could not count a newsletter message as {:?}: {}", id, fate, e);
905 }
906 Ok(())
907 })
908}
909
910/// Queues an entry again, changed, and removes the one it replaces. The new one is written first,
911/// so a stop between the two sends twice at worst and never loses a message. Where the one it replaces
912/// is already gone, nothing is queued.
913pub fn retry<
914 const UIDL: usize,
915 UID: NumIdDat<UIDL>,
916 ENC: Encrypter,
917 KH: Hasher,
918 DB: Database<UIDL, UID, ENC, KH>,
919>(
920 db: &(Arc<RwLock<DB>>, UID),
921 seq: u64,
922 next: &Entry,
923)
924 -> Outcome<()>
925{
926 store::exclusive(db, |dbr, user| -> Outcome<()> {
927 // An entry an erasure has taken off the queue is not put back.
928 if !res!(queued_at_in(dbr, next.kind, seq)) {
929 return Ok(());
930 }
931 res!(push_in(dbr, user, next));
932 pop_in(dbr, user, next.kind, seq)
933 })
934}
935
936// Records on the queued entry whether its message has gone whole (see [`Dial::keeping`]), so a drainer
937// stopped while the answer is awaited finds it after a restart and does not send it again.
938fn keep_whole<
939 const UIDL: usize,
940 UID: NumIdDat<UIDL>,
941 ENC: Encrypter,
942 KH: Hasher,
943 DB: Database<UIDL, UID, ENC, KH>,
944>(
945 db: &(Arc<RwLock<DB>>, UID),
946 entry: &Entry,
947 seq: u64,
948 whole: bool,
949)
950 -> Outcome<()>
951{
952 store::exclusive(db, |dbr, user| -> Outcome<()> {
953 if res!(queued_at_in(dbr, entry.kind, seq)) {
954 let e = Entry { whole, ..entry.clone() };
955 res!(dbr.insert(entry_key(entry.kind, seq), e.to_dat(), user, None));
956 }
957 Ok(())
958 })
959}
960
961
962// ┌───────────────────────────────────────────────────────────────────────────┐
963// │ DRAINER │
964// └───────────────────────────────────────────────────────────────────────────┘
965
966/// What one pass of the drainer did.
967#[derive(Clone, Copy, Debug, Eq, PartialEq)]
968pub enum Step {
969 Sent, // a message was accepted by the far end; call again at once
970 Worked, // an entry was skipped, refused, retried or given up; call again at once
971 Wait(u64), // the next slot is this many milliseconds away
972 Later(u64), // every entry is backing off; the soonest is this many seconds away
973 Idle, // nothing is queued
974 Held, // the ceiling is 0
975}
976
977/// Takes the next due entry, in lane order, and sends it if the pacer allows.
978///
979/// `now_ms` is the clock, so a test can run an hour in an instant. The pacer is asked only after the
980/// entry has proved still worth sending, so a skipped entry uses no slot, and an entry whose mail
981/// exchanges have all timed out is put back until the earliest block ends, using none. `share`
982/// decides which lane is
983/// looked at first. Nothing here holds the
984/// database lock across an `.await`, and the self-locking `subscribe` calls are made outside any
985/// [`store::exclusive`].
986pub async fn step<
987 const UIDL: usize,
988 UID: NumIdDat<UIDL>,
989 ENC: Encrypter,
990 KH: Hasher,
991 DB: Database<UIDL, UID, ENC, KH>,
992 C: Courier,
993>(
994 db: &(Arc<RwLock<DB>>, UID),
995 cfg: &PublishConfig,
996 courier: &C,
997 pacer: &Pacer,
998 share: &mut Share,
999 id: &str,
1000 now_ms: u64,
1001)
1002 -> Outcome<Step>
1003{
1004 if pacer.is_held() {
1005 return Ok(Step::Held);
1006 }
1007 let now = now_ms / 1000;
1008 let mut soonest: Option<u64> = None;
1009 let mut found: Option<(u64, Entry)> = None;
1010 for kind in share.order() {
1011 let mut left = u64::MAX;
1012 let mut sooner: Option<u64> = None;
1013 loop {
1014 match res!(take(db, kind, now, left)) {
1015 Take::Due(seq, e) => {
1016 found = Some((seq, e));
1017 break;
1018 }
1019 Take::More(t, rest) => {
1020 sooner = earliest(sooner, t);
1021 left = rest;
1022 // The guard is down between the parts; a moment lets the writers that waited in.
1023 tokio::time::sleep(Duration::from_millis(1)).await;
1024 }
1025 Take::Later(t) => {
1026 sooner = earliest(sooner, Some(t));
1027 break;
1028 }
1029 Take::Empty => break,
1030 }
1031 }
1032 if found.is_some() {
1033 break;
1034 }
1035 soonest = earliest(soonest, sooner);
1036 }
1037 let (seq, entry) = match found {
1038 Some(f) => f,
1039 None => return Ok(match soonest {
1040 Some(t) => Step::Later(t.saturating_sub(now)),
1041 None => Step::Idle,
1042 }),
1043 };
1044 // Its message went whole to a server before the drainer stopped, and no answer was recorded: it may
1045 // have been taken, so it is counted sent and never sent again.
1046 if entry.whole {
1047 warn!("{}: publish: {} may have the message from before a stop, counted unconfirmed",
1048 id, domain_of(&entry.email));
1049 match entry.kind {
1050 Kind::Confirm => res!(confirmed(db, cfg, &subscribe::normalise_email(&entry.email), seq, now, id, false)),
1051 Kind::News => res!(finish(db, &entry, seq, Fate::Unconfirmed, id)),
1052 }
1053 return Ok(Step::Worked);
1054 }
1055 match entry.kind {
1056 Kind::Confirm => confirm_step(db, cfg, courier, pacer, share, id, now_ms, seq, entry).await,
1057 Kind::News => news_step(db, cfg, courier, pacer, share, id, now_ms, seq, entry).await,
1058 }
1059}
1060
1061fn earliest(a: Option<u64>, b: Option<u64>) -> Option<u64> {
1062 match (a, b) {
1063 (Some(x), Some(y)) => Some(x.min(y)),
1064 (x, None) => x,
1065 (None, y) => y,
1066 }
1067}
1068
1069async fn confirm_step<
1070 const UIDL: usize,
1071 UID: NumIdDat<UIDL>,
1072 ENC: Encrypter,
1073 KH: Hasher,
1074 DB: Database<UIDL, UID, ENC, KH>,
1075 C: Courier,
1076>(
1077 db: &(Arc<RwLock<DB>>, UID),
1078 cfg: &PublishConfig,
1079 courier: &C,
1080 pacer: &Pacer,
1081 share: &mut Share,
1082 id: &str,
1083 now_ms: u64,
1084 seq: u64,
1085 entry: Entry,
1086)
1087 -> Outcome<Step>
1088{
1089 let now = now_ms / 1000;
1090 let w = cfg.confirm_window();
1091 let email = subscribe::normalise_email(&entry.email);
1092 if !subscribe::valid_email(&email) {
1093 warn!("{}: publish: dropping a sign-up that names no address", id);
1094 res!(done(db, entry.kind, seq));
1095 return Ok(Step::Worked);
1096 }
1097 // An address whose exchanges have all timed out is left alone, and costs no slot and no try until
1098 // the earliest block ends.
1099 let (dial, budget) = match res!(ahead(courier, pacer, &email, now_ms, id).await) {
1100 Ahead::Held => return Ok(Step::Held),
1101 Ahead::Wait(ms) => return Ok(Step::Wait(ms)),
1102 Ahead::Blocked(until) => return defer(db, seq, entry, until, id),
1103 Ahead::Clear(dial, t) => (dial, t),
1104 };
1105 let kept = entry.clone();
1106 let dial = dial.keeping(move |whole| keep_whole(db, &kept, seq, whole));
1107 // The slot is taken for every entry, before the sign-up is looked at. An entry that ends unsent (a
1108 // repeat, or an address that has confirmed, left or bounced) takes the same gap as one that is sent,
1109 // so when the entries behind it leave says nothing about whether the address was on the list.
1110 match res!(pacer.claim(now_ms)) {
1111 Pace::Held => return Ok(Step::Held),
1112 Pace::Wait(ms) => return Ok(Step::Wait(ms)),
1113 Pace::Go => share.took(Kind::Confirm),
1114 }
1115 // The entry names an address and nothing else: the sign-up is applied here, at send time, so an
1116 // address that has confirmed, left or bounced since is owed nothing, one already sent a
1117 // confirmation within its interval or its count is owed no other, and the token in the link is
1118 // whatever the record holds now. A repeat sign-up queued behind the first ends here, and so does one
1119 // an erasure removed after it was taken, which is applied only while its entry is still queued.
1120 let sub = match res!(subscribe::add_pending_from(db, &email, &w, now, Some((entry.kind, seq)))) {
1121 Some(s) => s,
1122 None => {
1123 debug!("{}: publish: sign-up of {} owed no confirmation", id, subscribe::redact(&email));
1124 res!(done(db, entry.kind, seq));
1125 return Ok(Step::Worked);
1126 }
1127 };
1128 let from = cfg.from_or(courier.default_from());
1129 let url = cfg.url_of(&cfg.confirm_path(&sub.token));
1130 let msg = send::build_confirmation_email(&from, &sub.email, &url, &cfg.site_name);
1131 let mut step = Step::Worked;
1132 match send_within(courier, pacer, id, &from, &sub.email, &msg, now, &dial, budget).await {
1133 Ok(qid) => {
1134 step = Step::Sent;
1135 info!("{}: publish: confirmation sent to {}", id, subscribe::redact(&sub.email));
1136 res!(confirmed(db, cfg, &sub.email, seq, now, id, qid.is_some()));
1137 }
1138 // A permanent failure means the address does not exist; suppress it so no later sign-up
1139 // mails a mailbox the remote has refused.
1140 Err(e) if is_permanent(&e) => {
1141 warn!("{}: publish: confirmation to {} failed permanently; suppressing: {}",
1142 id, subscribe::redact(&sub.email), e);
1143 res!(done(db, entry.kind, seq));
1144 if let Err(e2) = subscribe::mark_bounced(db, &sub.email, id) {
1145 warn!("{}: publish: could not suppress {}: {}",
1146 id, subscribe::redact(&sub.email), e2);
1147 }
1148 }
1149 Err(e) => {
1150 let tries = entry.tries + 1;
1151 if tries >= TRIES_MAX {
1152 warn!("{}: publish: confirmation to {} given up after {} tries: {}",
1153 id, subscribe::redact(&sub.email), tries, e);
1154 res!(done(db, entry.kind, seq));
1155 } else {
1156 warn!("{}: publish: confirmation to {} did not send, try {}: {}",
1157 id, subscribe::redact(&sub.email), tries, e);
1158 let next = Entry {
1159 tries,
1160 next_try: now + retry_after(tries),
1161 ..entry
1162 };
1163 res!(retry(db, seq, &next));
1164 }
1165 }
1166 }
1167 Ok(step)
1168}
1169
1170// A confirmation that has left, or may have: off the queue, then recorded. Only one `taken` is counted
1171// against the address: one that may not have arrived leaves the person free to ask again, since a
1172// second copy is better than none.
1173fn confirmed<
1174 const UIDL: usize,
1175 UID: NumIdDat<UIDL>,
1176 ENC: Encrypter,
1177 KH: Hasher,
1178 DB: Database<UIDL, UID, ENC, KH>,
1179>(
1180 db: &(Arc<RwLock<DB>>, UID),
1181 cfg: &PublishConfig,
1182 email: &str,
1183 seq: u64,
1184 now: u64,
1185 id: &str,
1186 taken: bool,
1187)
1188 -> Outcome<()>
1189{
1190 // Gone first, so a failure after the send can never send it twice.
1191 res!(done(db, Kind::Confirm, seq));
1192 let at = send::iso_of(now as i64).unwrap_or_default();
1193 if let Err(e) = subscribe::mark_sent(db, email, &at) {
1194 warn!("{}: publish: could not record the send to {}: {}", id, subscribe::redact(email), e);
1195 }
1196 if !taken {
1197 return Ok(());
1198 }
1199 if let Err(e) = subscribe::count_sent(db, email, &cfg.confirm_window(), now) {
1200 warn!("{}: publish: could not count the send to {}: {}", id, subscribe::redact(email), e);
1201 }
1202 Ok(())
1203}
1204
1205// The domain of a normalised address; empty where it has none.
1206fn domain_of(email: &str) -> &str {
1207 email.rsplit_once('@').map(|(_, d)| d).unwrap_or("")
1208}
1209
1210// Puts an entry back, not to be tried before `until`, because its mail exchanges are not answering.
1211// Nothing was sent, so no slot is used and no try is counted.
1212fn defer<
1213 const UIDL: usize,
1214 UID: NumIdDat<UIDL>,
1215 ENC: Encrypter,
1216 KH: Hasher,
1217 DB: Database<UIDL, UID, ENC, KH>,
1218>(
1219 db: &(Arc<RwLock<DB>>, UID),
1220 seq: u64,
1221 entry: Entry,
1222 until: u64,
1223 id: &str,
1224)
1225 -> Outcome<Step>
1226{
1227 debug!("{}: publish: {} put back until {}, its mail exchanges are not answering",
1228 id, subscribe::redact(&entry.email), until);
1229 let next = Entry { next_try: until, ..entry };
1230 res!(retry(db, seq, &next));
1231 Ok(Step::Worked)
1232}
1233
1234// What the drainer learns before it takes a slot for an entry.
1235enum Ahead {
1236 Held, // the ceiling is 0
1237 Wait(u64), // the next slot is this many milliseconds away
1238 Blocked(u64), // every exchange has timed out; the earliest block ends at this unix second
1239 Clear(Dial<'static>, Duration), // send, keeping away from the exchanges in the dial, within this long
1240}
1241
1242// Looks at the exchanges of a recipient before the slot is claimed for it. The slot is asked after, so
1243// nothing is looked up for a send that would have to wait, and a blocked entry is put back without the
1244// slot being spent. The courier is asked under the deadline, and what it cannot answer in time, or at
1245// all, leaves the exchanges unknown: the send goes ahead as it would have, avoiding none. What the
1246// lookup used comes off the time the delivery is given, so the pair together stay within one deadline.
1247async fn ahead<C: Courier>(
1248 courier: &C,
1249 pacer: &Pacer,
1250 to: &str,
1251 now_ms: u64,
1252 id: &str,
1253)
1254 -> Outcome<Ahead>
1255{
1256 match res!(pacer.ready(now_ms)) {
1257 Pace::Held => return Ok(Ahead::Held),
1258 Pace::Wait(ms) => return Ok(Ahead::Wait(ms)),
1259 Pace::Go => {},
1260 }
1261 let deadline = pacer.deadline();
1262 let began = Instant::now();
1263 let ips = match tokio::time::timeout(deadline, courier.exchanges(to)).await {
1264 Ok(Ok(ips)) => ips,
1265 Ok(Err(e)) => {
1266 debug!("{}: publish: could not find the exchanges of {}: {}", id, domain_of(to), e);
1267 Vec::new()
1268 }
1269 Err(_) => {
1270 warn!("{}: publish: finding the exchanges of {} took more than {:?}",
1271 id, domain_of(to), deadline);
1272 return Ok(Ahead::Clear(Dial::default(), deadline));
1273 }
1274 };
1275 let budget = deadline.saturating_sub(began.elapsed());
1276 let blocks = res!(pacer.blocks(&ips, now_ms / 1000));
1277 if !ips.is_empty() && ips.iter().all(|ip| blocks.iter().any(|b| b.0 == *ip)) {
1278 let until = blocks.iter().map(|b| b.1).min().unwrap_or(0);
1279 return Ok(Ahead::Blocked(until));
1280 }
1281 let avoid: Vec<IpAddr> = blocks.iter().map(|b| b.0).collect();
1282 Ok(Ahead::Clear(Dial::avoiding(avoid), budget))
1283}
1284
1285// The exchange a delivery last dialled, as its dial noted it.
1286fn dialled(dial: &Dial, id: &str) -> Option<IpAddr> {
1287 match dial.last() {
1288 Ok(ip) => ip,
1289 Err(e) => {
1290 warn!("{}: publish: could not read the exchange dialled: {}", id, e);
1291 None
1292 }
1293 }
1294}
1295
1296// Delivers one message, giving up when `budget` passes so that one far end that does not answer cannot
1297// hold the drainer. An overrun is a transient failure like any other, and it also blocks the exchange
1298// the delivery was waiting on (see [`Pacer::timed_out`]); where the courier had dialled none, there is
1299// none to block. A delivery the far end answered, accepting or refusing outright, clears the exchange
1300// it reached. The bookkeeping never changes the outcome: a message that has left must be reported as
1301// sent, with the far end's queue id, or with none where the far end may have it and never said.
1302async fn send_within<C: Courier>(
1303 courier: &C,
1304 pacer: &Pacer,
1305 id: &str,
1306 from: &str,
1307 to: &str,
1308 msg: &str,
1309 now: u64,
1310 dial: &Dial<'_>,
1311 budget: Duration,
1312)
1313 -> Outcome<Option<String>>
1314{
1315 let domain = domain_of(to);
1316 match tokio::time::timeout(budget, courier.deliver(from, to, msg, dial)).await {
1317 Ok(r) => {
1318 let answered = match &r {
1319 Ok(_) => true,
1320 Err(e) => is_permanent(e) || maybe_sent(e),
1321 };
1322 if answered {
1323 if let Some(ip) = dialled(dial, id) {
1324 if let Err(e) = pacer.reached(ip) {
1325 warn!("{}: publish: could not clear {} after a delivery: {}", id, ip, e);
1326 }
1327 }
1328 }
1329 match r {
1330 // The exchange may have it: counted unconfirmed, so neither sent again nor suppressed.
1331 Err(e) if maybe_sent(&e) || dial.whole() => {
1332 warn!("{}: publish: {} may have the message, counted unconfirmed: {}", id, domain, e);
1333 Ok(None)
1334 }
1335 r => r.map(Some),
1336 }
1337 }
1338 Err(_) => {
1339 match dialled(dial, id) {
1340 Some(ip) => {
1341 let until = match pacer.timed_out(ip, now) {
1342 Ok(t) => t,
1343 Err(e) => {
1344 warn!("{}: publish: could not block {} after a timeout: {}", id, ip, e);
1345 now
1346 }
1347 };
1348 warn!("{}: publish: the exchange {} of {} did not answer within {:?}; left alone until {}",
1349 id, ip, domain, budget, until);
1350 }
1351 None => warn!("{}: publish: a delivery to {} dialled nothing within {:?}",
1352 id, domain, budget),
1353 }
1354 // Given up on while the answer to the whole message was awaited: the far end may have it.
1355 if dial.whole() {
1356 warn!("{}: publish: {} may have the message, counted unconfirmed: no answer within {:?}",
1357 id, domain, budget);
1358 return Ok(None);
1359 }
1360 Err(err!("publish: a delivery to {} did not finish within {:?}.", domain, pacer.deadline();
1361 Timeout, Network))
1362 }
1363 }
1364}
1365
1366// A newsletter message that is no longer owed: the post is no longer live, or the subscriber has gone.
1367// It leaves the queue and the send set, and uses no slot.
1368fn left<
1369 const UIDL: usize,
1370 UID: NumIdDat<UIDL>,
1371 ENC: Encrypter,
1372 KH: Hasher,
1373 DB: Database<UIDL, UID, ENC, KH>,
1374>(
1375 db: &(Arc<RwLock<DB>>, UID),
1376 entry: &Entry,
1377 seq: u64,
1378 id: &str,
1379)
1380 -> Outcome<Step>
1381{
1382 debug!("{}: publish: newsletter '{}' to {} skipped, no longer owed",
1383 id, entry.slug, subscribe::redact(&entry.email));
1384 res!(finish(db, entry, seq, Fate::Left, id));
1385 Ok(Step::Worked)
1386}
1387
1388async fn news_step<
1389 const UIDL: usize,
1390 UID: NumIdDat<UIDL>,
1391 ENC: Encrypter,
1392 KH: Hasher,
1393 DB: Database<UIDL, UID, ENC, KH>,
1394 C: Courier,
1395>(
1396 db: &(Arc<RwLock<DB>>, UID),
1397 cfg: &PublishConfig,
1398 courier: &C,
1399 pacer: &Pacer,
1400 share: &mut Share,
1401 id: &str,
1402 now_ms: u64,
1403 seq: u64,
1404 entry: Entry,
1405)
1406 -> Outcome<Step>
1407{
1408 let now = now_ms / 1000;
1409 // Re-read at send time: a post unpublished, or a subscriber who has left, since the entry was
1410 // queued is owed nothing, and the unsubscribe link carries the token the record holds now.
1411 let rec = match res!(store::get(db, &entry.slug)) {
1412 Some(r) if r.state == PostState::Live => r,
1413 _ => return left(db, &entry, seq, id),
1414 };
1415 let sub = match res!(subscribe::get(db, &entry.email)) {
1416 Some(s) if s.state == SubState::Confirmed => s,
1417 _ => return left(db, &entry, seq, id),
1418 };
1419 // Rendered before the slot is claimed, so a post that will not render costs no slot.
1420 let post = match rec.render() {
1421 Ok(p) => p,
1422 Err(e) => {
1423 warn!("{}: publish: newsletter '{}' will not render, so {} is not sent it: {}",
1424 id, entry.slug, subscribe::redact(&entry.email), e);
1425 res!(finish(db, &entry, seq, Fate::Failed, id));
1426 return Ok(Step::Worked);
1427 }
1428 };
1429 // An address whose exchanges have all timed out is left alone, and costs no slot and no try until
1430 // the earliest block ends.
1431 let (dial, budget) = match res!(ahead(courier, pacer, &sub.email, now_ms, id).await) {
1432 Ahead::Held => return Ok(Step::Held),
1433 Ahead::Wait(ms) => return Ok(Step::Wait(ms)),
1434 Ahead::Blocked(until) => return defer(db, seq, entry, until, id),
1435 Ahead::Clear(dial, t) => (dial, t),
1436 };
1437 let kept = entry.clone();
1438 let dial = dial.keeping(move |whole| keep_whole(db, &kept, seq, whole));
1439 match res!(pacer.claim(now_ms)) {
1440 Pace::Held => return Ok(Step::Held),
1441 Pace::Wait(ms) => return Ok(Step::Wait(ms)),
1442 Pace::Go => share.took(Kind::News),
1443 }
1444 let from = cfg.from_or(courier.default_from());
1445 let online = cfg.url_of(&cfg.path_of(&entry.slug));
1446 let unsub = cfg.url_of(&cfg.unsubscribe_path(&sub.token));
1447 let msg = send::build_newsletter_email(&from, &sub.email, &post, &online, &unsub, &cfg.site_name);
1448 let mut step = Step::Worked;
1449 match send_within(courier, pacer, id, &from, &sub.email, &msg, now, &dial, budget).await {
1450 Ok(qid) => {
1451 step = Step::Sent;
1452 debug!("{}: publish: newsletter '{}' to {} ({:?})",
1453 id, entry.slug, subscribe::redact(&sub.email), qid);
1454 let fate = match qid {
1455 Some(_) => Fate::Sent,
1456 None => Fate::Unconfirmed,
1457 };
1458 // Gone first, so a failure after the send can never send it twice.
1459 res!(finish(db, &entry, seq, fate, id));
1460 }
1461 // A permanent failure suppresses the address so no future send reaches it.
1462 Err(e) if is_permanent(&e) => {
1463 warn!("{}: publish: newsletter '{}' to {} failed permanently; suppressing: {}",
1464 id, entry.slug, subscribe::redact(&sub.email), e);
1465 res!(finish(db, &entry, seq, Fate::Suppressed, id));
1466 if let Err(e2) = subscribe::mark_bounced(db, &sub.email, id) {
1467 warn!("{}: publish: could not suppress {}: {}",
1468 id, subscribe::redact(&sub.email), e2);
1469 }
1470 }
1471 Err(e) => {
1472 let tries = entry.tries + 1;
1473 if tries >= TRIES_MAX {
1474 warn!("{}: publish: newsletter '{}' to {} given up after {} tries: {}",
1475 id, entry.slug, subscribe::redact(&sub.email), tries, e);
1476 res!(finish(db, &entry, seq, Fate::Failed, id));
1477 } else {
1478 warn!("{}: publish: newsletter '{}' to {} did not send, try {}: {}",
1479 id, entry.slug, subscribe::redact(&sub.email), tries, e);
1480 let next = Entry {
1481 tries,
1482 next_try: now + retry_after(tries),
1483 ..entry
1484 };
1485 res!(retry(db, seq, &next));
1486 }
1487 }
1488 }
1489 Ok(step)
1490}
1491
1492
1493// ┌───────────────────────────────────────────────────────────────────────────┐
1494// │ WATCH │
1495// └───────────────────────────────────────────────────────────────────────────┘
1496
1497/// What the queue holds, as the alert reads it.
1498#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
1499pub struct Stats {
1500 pub queued: u64, // entries in every lane, including those backing off
1501 pub oldest: Option<u64>, // unix second the oldest lane head was first queued; None if empty
1502}
1503
1504/// Reads the queue's depth and the age of what waits longest at the head of a lane.
1505///
1506/// An entry retried goes to the tail but keeps the time it was first queued, so a lane's head is its
1507/// oldest entry except while one is backing off, which [`retry_after`] keeps well inside any alert
1508/// threshold worth setting.
1509pub fn stats<
1510 const UIDL: usize,
1511 UID: NumIdDat<UIDL>,
1512 ENC: Encrypter,
1513 KH: Hasher,
1514 DB: Database<UIDL, UID, ENC, KH>,
1515>(
1516 db: &(Arc<RwLock<DB>>, UID),
1517)
1518 -> Outcome<Stats>
1519{
1520 let (db_arc, _) = db;
1521 let guard = lock_read!(db_arc);
1522 let mut out = Stats::default();
1523 for kind in KINDS {
1524 let head = res!(counter_in(&*guard, &head_key(kind)));
1525 let tail = res!(counter_in(&*guard, &tail_key(kind)));
1526 if tail <= head {
1527 continue;
1528 }
1529 out.queued += tail - head;
1530 let end = tail.min(head.saturating_add(HEAD_LOOK));
1531 for at in head..end {
1532 if let Ok(Some(e)) = entry_in(&*guard, kind, at) {
1533 out.oldest = Some(out.oldest.map_or(e.enqueued, |o| o.min(e.enqueued)));
1534 break;
1535 }
1536 }
1537 }
1538 Ok(out)
1539}
1540
1541/// Tells one backlog episode apart from the next, so the operator hears of each once.
1542///
1543/// An episode begins when the oldest entry has waited longer than the threshold, is told again each
1544/// day it lasts, and ends only when the queue has drained. A queue that stays deep has not recovered
1545/// because its oldest entry's age dipped under the threshold, and ending the episode there told the
1546/// operator again at every crossing. It lives in memory, so a restart during a backlog begins a new
1547/// episode and tells it again.
1548#[derive(Clone, Debug, Default)]
1549pub struct Watch {
1550 since: Option<u64>, // unix second the episode was first told
1551 told: u64, // unix second of the latest telling
1552 sent: u64, // messages sent since the episode began
1553}
1554
1555impl Watch {
1556
1557 /// Counts a message sent, if an episode is open.
1558 pub fn sent(&mut self) {
1559 if self.since.is_some() {
1560 self.sent += 1;
1561 }
1562 }
1563
1564 /// What, if anything, the operator is owed at `now`, given what the queue holds. A `limit` of 0
1565 /// turns the alert off.
1566 pub fn note(&mut self, site: &str, st: &Stats, hourly: u32, limit: u64, now: u64)
1567 -> Option<AlertEvent>
1568 {
1569 let waited = st.oldest.map_or(0, |o| now.saturating_sub(o));
1570 let over = limit > 0 && st.queued > 0 && waited > limit;
1571 match (over, self.since) {
1572 (true, None) => {
1573 self.since = Some(now);
1574 self.told = now;
1575 self.sent = 0;
1576 Some(AlertEvent::OutboxBacklog {
1577 site: site.to_string(),
1578 queued: st.queued,
1579 oldest_secs: waited,
1580 hourly,
1581 })
1582 }
1583 (true, Some(_)) => {
1584 if now.saturating_sub(self.told) < DAY_SECS {
1585 return None;
1586 }
1587 self.told = now;
1588 Some(AlertEvent::OutboxBacklog {
1589 site: site.to_string(),
1590 queued: st.queued,
1591 oldest_secs: waited,
1592 hourly,
1593 })
1594 }
1595 (false, Some(since)) if st.queued == 0 => {
1596 let sent = self.sent;
1597 *self = Self::default();
1598 Some(AlertEvent::OutboxCleared {
1599 site: site.to_string(),
1600 were_secs: now.saturating_sub(since),
1601 sent,
1602 })
1603 }
1604 (false, _) => None,
1605 }
1606 }
1607}
1608
1609/// One pass of the drainer, then a look at the queue: the step it took and the alert it owes.
1610pub async fn tick<
1611 const UIDL: usize,
1612 UID: NumIdDat<UIDL>,
1613 ENC: Encrypter,
1614 KH: Hasher,
1615 DB: Database<UIDL, UID, ENC, KH>,
1616 C: Courier,
1617>(
1618 db: &(Arc<RwLock<DB>>, UID),
1619 cfg: &PublishConfig,
1620 courier: &C,
1621 pacer: &Pacer,
1622 share: &mut Share,
1623 watch: &mut Watch,
1624 id: &str,
1625 now_ms: u64,
1626)
1627 -> Outcome<(Step, Option<AlertEvent>)>
1628{
1629 let stepped = res!(step(db, cfg, courier, pacer, share, id, now_ms).await);
1630 if stepped == Step::Sent {
1631 watch.sent();
1632 }
1633 let st = res!(stats(db));
1634 let owed = watch.note(id, &st, pacer.hourly(), cfg.outbox_alert_secs, now_ms / 1000);
1635 Ok((stepped, owed))
1636}
1637
1638// ┌───────────────────────────────────────────────────────────────────────────┐
1639// │ SWEEP │
1640// └───────────────────────────────────────────────────────────────────────────┘
1641
1642/// What one sweep removed.
1643#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
1644pub struct Swept {
1645 pub pending: usize, // unconfirmed sign-ups past `pending_expiry_days`
1646 pub rows: usize, // rate rows whose window has passed
1647}
1648
1649/// Removes what has outlived its use: sign-ups that never confirmed, and rate rows whose window has
1650/// passed. Run once a day by the drainer, never by a request.
1651///
1652/// The two halves are independent, so one failing is logged and does not stop the other. The rows
1653/// need a scan of the whole store (see [`rate::expire_rows`]), which is why this is a daily sweep
1654/// and not work done as each row is touched. `sending` is whether the host's ceiling is above 0; see
1655/// [`subscribe::expire_pending`].
1656pub fn sweep<
1657 const UIDL: usize,
1658 UID: NumIdDat<UIDL>,
1659 ENC: Encrypter,
1660 KH: Hasher,
1661 DB: Database<UIDL, UID, ENC, KH>,
1662>(
1663 db: &(Arc<RwLock<DB>>, UID),
1664 cfg: &PublishConfig,
1665 now: u64,
1666 id: &str,
1667 sending: bool,
1668)
1669 -> Swept
1670{
1671 let pending = match subscribe::expire_pending(db, cfg.pending_expiry_days, now, id, sending) {
1672 Ok(n) => n,
1673 Err(e) => {
1674 warn!("{}: publish: expiring unconfirmed sign-ups failed: {}", id, e);
1675 0
1676 }
1677 };
1678 let sets = [
1679 (subscribe::RATE_PREFIX, Window::hourly(cfg.subscribe_rate_secs, cfg.subscribe_rate_hourly)),
1680 (subscribe::TO_PREFIX, cfg.confirm_window()),
1681 (comment::RATE_PREFIX, Window::hourly(cfg.comment_rate_secs, cfg.comment_rate_hourly)),
1682 ];
1683 let rows = match rate::expire_rows(db, &sets, now) {
1684 Ok(n) => n,
1685 Err(e) => {
1686 warn!("{}: publish: expiring spent rate rows failed: {}", id, e);
1687 0
1688 }
1689 };
1690 if rows > 0 {
1691 info!("{}: publish: {} spent rate row(s) expired", id, rows);
1692 }
1693 Swept { pending, rows }
1694}
1695
1696
1697// Tells the operator, or says in the log that nobody can be told.
1698fn tell(alerter: &Option<Alerter>, id: &str, event: AlertEvent) {
1699 match alerter {
1700 Some(a) => a.raise(event),
1701 None => warn!("{}: publish: {} (no alerter is configured, so nobody is told)",
1702 id, event.subject(id)),
1703 }
1704}
1705
1706/// Drains a vhost's queue for as long as the server runs.
1707///
1708/// Sleeps until the next slot, or until something is queued, or for [`POLL_SECS`], whichever is
1709/// first, and tells the operator through `alerter` what the [`Watch`] finds. A pass that fails is logged and tried again after the poll interval, never in a tight
1710/// loop.
1711///
1712/// Once a day, and once at the start, it also runs [`sweep`] on a blocking thread, since the sweep
1713/// scans the store. The bounds are `'static` only for that thread.
1714pub async fn run<
1715 const UIDL: usize,
1716 UID: NumIdDat<UIDL> + 'static,
1717 ENC: Encrypter + 'static,
1718 KH: Hasher + 'static,
1719 DB: Database<UIDL, UID, ENC, KH> + 'static,
1720 C: Courier,
1721>(
1722 db: (Arc<RwLock<DB>>, UID),
1723 cfg: Arc<PublishConfig>,
1724 courier: Arc<C>,
1725 pacer: Arc<Pacer>,
1726 alerter: Option<Alerter>,
1727 id: String,
1728) {
1729 info!("{}: publish: the outbox drainer is running at {} an hour", id, pacer.hourly());
1730 let mut watch = Watch::default();
1731 let mut share = Share::default();
1732 let mut swept: Option<u64> = None;
1733 loop {
1734 let now = now_ms() / 1000;
1735 if swept.map_or(true, |t| now.saturating_sub(t) >= DAY_SECS) {
1736 // Marked first, so a sweep that fails is tried again tomorrow and not on every pass.
1737 swept = Some(now);
1738 let (d, c, i, sending) = (db.clone(), cfg.clone(), id.clone(), !pacer.is_held());
1739 if let Err(e) = tokio::task::spawn_blocking(move || sweep(&d, &c, now, &i, sending)).await {
1740 warn!("{}: publish: the daily sweep did not finish: {}", id, e);
1741 }
1742 }
1743 // Registered before the pass, so a push during it is not missed.
1744 let woken = pacer.wake.notified();
1745 tokio::pin!(woken);
1746 woken.as_mut().enable();
1747 let nap_ms = match tick(&db, &cfg, &*courier, &pacer, &mut share, &mut watch, &id, now_ms()).await {
1748 Ok((stepped, owed)) => {
1749 if let Some(event) = owed {
1750 tell(&alerter, &id, event);
1751 }
1752 match stepped {
1753 Step::Sent | Step::Worked => continue,
1754 Step::Wait(ms) => ms,
1755 Step::Later(s) => s.min(POLL_SECS).saturating_mul(1000),
1756 Step::Idle | Step::Held => POLL_SECS * 1000,
1757 }
1758 }
1759 Err(e) => {
1760 warn!("{}: publish: the outbox drainer failed: {}", id, e);
1761 POLL_SECS * 1000
1762 }
1763 };
1764 tokio::select! {
1765 _ = &mut woken => {},
1766 _ = tokio::time::sleep(Duration::from_millis(nap_ms.max(1))) => {},
1767 }
1768 }
1769}