How do you cope with distributed transactions? (I'm not sure if you support them?)
I know that Citus has a `citus_create_restore_point()` (or so) function that, when called, guarantees that no 2PC commits are in flight and creates a WAL restore point in every shard. Therefore, restoring shards to that point will leave the DB in a consistent state. Do you do something similar?
Neki will support cross-shard ACID transactions with a combination of an external transaction coordinator and some changes to PostgreSQL itself to support this (either by engine modifications or extensions).
I expect as part of that, we'll allow users to leverage something similar to make sure that we don't get transaction tearing in backups.
PostgresTV livestreams "vibe coding" 1h sessions implementing small PostgreSQL features with Cursor (mostly claude-4-sonnet model) every week, by experienced PostgreSQL contributors. [0] is their latest stream.
I personally have not watched much, but it sounds just like what you are looking for!
Headless browsers consume orders of magnitude more resources, and execute far more requests (e.g. fetching images) than a common webscraping job would require. Having run webscraping at scale myself, the cost of operating headless browsers made us only use them as a last resort.
Speaking as a person who has played on both offense and defense: this is a heuristic that's not used frequently enough by defenders. Clients that load a single HTML/JSON endpoint without loading css or image resources associated with the endpoints are likely bots (or user agents with a fully loaded cache, but defenders control what gets cached by legit clients and how). Bot data thriftiness is a huge signal.
Even legitimate users might want to disable CSS and pictures and whatever, and I often do when I just want to read the document.
Blind users also might have no use for the pictures, and another possibility is if the document is longer than the screen so the picture is out of view then the user might program the client software to use lazy loading, etc.
I know that Citus has a `citus_create_restore_point()` (or so) function that, when called, guarantees that no 2PC commits are in flight and creates a WAL restore point in every shard. Therefore, restoring shards to that point will leave the DB in a consistent state. Do you do something similar?