Add pseudo listen/notify mechanism for SQLite#1275
Draft
brandur wants to merge 1 commit into
Draft
Conversation
37fe29c to
44e8909
Compare
1f2087b to
775c6f0
Compare
A fairly long-standing issue with SQLite compared to Postgres is the lack of a listen/notify system because SQLite doesn't offer such a mechanism. This has forced SQLite to operate in poll-only mode which means there's a longer delay before it sees jobs that it can work. Honker [1] hit Hacker News a few weeks/months back. It implements a listen/notify-like mechanism, and when reading about how it's implemented, I realized it wouldn't be all that crazy to do something similar for River. Basically, it's poll based, and relies on polling a single table fairly actively. That's something that's not great in general, but we already do that for poll-only, except that it's multiple tables instead of just one (i.e. `river_job` + `river_leader`). Here, I propose that we implement listen/notify using a similar strategy by adding a `river_notification` in SQLite. Like listen/notify, this table has a listen/notify, along with a timestamp that we can use to track event creation and prune older rows. We add a SQLite `Listener` similar to the one for Postgres, but which polls `river_notification`, remembering the last ID that it so it can make sure to only fetch new notifications in each River process. A new maintenance service that runs only in SQLite takes care of pruning events in the table older than 5 minutes. A nice property of `river_notification` is that like listen/notify, it all works transactionally as well. New rows are hidden until the transaction that inserted them commits, keeping them invisible from the listener. After commit, the listener sees them and responds. We track a `lastID` in the listener for more efficient reads, which seems like it might be a problem due to out-of-order IDs, but for better or worse this is not possible in SQLite due its single-writer model. IDs always commit in order. You might think that this would create a table that's overly high volume because 1,000 job insertions would create 1,000 notifications, but luckily we've been deduplicating notifications through the use of a notify limiter (i.e. `insertNotifyLimiter.ShouldTrigger(queue)`) for a long time now. Leadership notifications are also low volume, so this table should never have to be very big at any given time, which is nice. Requires a database migration, but that might dovetail fairly nicely with #1224 (convert SQLite json to jsonb), which also does. I've experimented with a few versions of this. This one also adds a `river_notification` to Postgres, though doesn't use it. The rationale here is (1) that the Postgres `river_notification` makes it easier to test the client as if it was SQLite but without having to bring a SQLite driver into the top-level project as a dependency, and (2) it may be a useful alternative to poll-only even in Postgres down the line, where something like a bouncer makes listen/notify difficult to use. [1] https://github.com/russellromney/honker
775c6f0 to
dfd504b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A fairly long-standing issue with SQLite compared to Postgres is the
lack of a listen/notify system because SQLite doesn't offer such a
mechanism. This has forced SQLite to operate in poll-only mode which
means there's a longer delay before it sees jobs that it can work.
Honker [1] hit Hacker News a few weeks/months back. It implements a
listen/notify-like mechanism, and when reading about how it's
implemented, I realized it wouldn't be all that crazy to do something
similar for River. Basically, it's poll based, and relies on polling a
single table fairly actively. That's something that's not great in
general, but we already do that for poll-only, except that it's multiple
tables instead of just one (i.e.
river_job+river_leader).Here, I propose that we implement listen/notify using a similar strategy
by adding a
river_notificationin SQLite. Like listen/notify, thistable has a listen/notify, along with a timestamp that we can use to
track event creation and prune older rows. We add a SQLite
Listenersimilar to the one for Postgres, but which polls
river_notification,remembering the last ID that it so it can make sure to only fetch new
notifications in each River process.
A new maintenance service that runs only in SQLite takes care of pruning
events in the table older than 5 minutes.
A nice property of
river_notificationis that like listen/notify, itall works transactionally as well. New rows are hidden until the
transaction that inserted them commits, keeping them invisible from the
listener. After commit, the listener sees them and responds. We track a
lastIDin the listener for more efficient reads, which seems like itmight be a problem due to out-of-order IDs, but for better or worse this
is not possible in SQLite due its single-writer model. IDs always commit
in order.
You might think that this would create a table that's overly high volume
because 1,000 job insertions would create 1,000 notifications, but
luckily we've been deduplicating notifications through the use of a
notify limiter (i.e.
insertNotifyLimiter.ShouldTrigger(queue)) for along time now. Leadership notifications are also low volume, so this
table should never have to be very big at any given time, which is nice.
Requires a database migration, but that might dovetail fairly nicely
with #1224 (convert SQLite json to jsonb), which also does.
I've experimented with a few versions of this. This one also adds a
river_notificationto Postgres, though doesn't use it. The rationalehere is (1) that the Postgres
river_notificationmakes it easier totest the client as if it was SQLite but without having to bring a SQLite
driver into the top-level project as a dependency, and (2) it may be a
useful alternative to poll-only even in Postgres down the line, where
something like a bouncer makes listen/notify difficult to use.
[1] https://github.com/russellromney/honker