You will possibly want to combine the data for some reason in the future as well. Or, move ownership of resources atomically.

I'm not opposed to this setup at all and it does have its place. But we are running away from schema-per-tenant setup at warp speed at work. There are so many issues if you don't invest in it properly and I don't think many are prepared when they initially have the idea.

The funny thing is that about a decade ago, the app was born on a SQLite per tenant setup, then it moved to schema per tenant on Postgres, now it's finally moving to a single schema with RLS. So, the exact opposite progression.

That shouldn't be a big issue. Any service large/complex enough to care does the schema upgrades in phases, so it's 1. Make code future compatible. 2. Migrate data. 3. Remove old schema support.

So typically it should be safe to run between steps 1 and 2 for a long time. (Modulo new bugs of course) As an ops-y person I'm comfortable with the system running mid-migration as long as the steps as described are used.

Exactly this, schema migrations should be an append, deprecate, drop operation over time.

definitely have shot myself in the foot with postgres on this

You can use a linter for PostgreSQL migrations https://squawkhq.com/

That can be a good thing if your product has say

I guess it depends on the business structure.

To be fair, RLS was not available yet a decade ago :) It appeared in PostgreSQL 9.5 in 2016.

https://electric-sql.com/

This can be accounted for and handled. Though if schema issues are enough of a scare I wonder if a documentdb style embedable database like a couch/pouchdb might make more sense.

I've been using this to update/read db from multiple threads/processes on the same machine. You can also do snapshotting with the sqlite backup API, if you want consistent view, and to not hold on transaction (or copy in-memory).

But maybe I'm missing something here... Also haven't touched sqlite in years, so not sure...

So stand corrected!

[1] https://sqlite.org/hctree/doc/hctree/doc/hctree/index.html

I think the OP meant that updates have to run sequentially.

And the opposite more frequent checkpoints, means faster reads (no need to through bigger wal file, and smaller index), but writes are slower.

So it really depends on what's happening right now, if you can anticipate it - e.g. populating for the first time data into it (maybe decrease checkpoint updates, then turn it back on).

Or if you constantly log, and read only that much (though not sure if you have constraints, triggers whether there are no hidden reads).

Or the opposite - a "read-only" if possible version of sqlite.db would be ideally without any wal.

So your post helped understand that there is stuff that I don't know and need to look further into it.

Thanks!

SQLite + Litestream is an even greater choice for tenant databases, that's vastly cheaper to replicate/backup to S3/R2 than expensive cloud managed databases [1] (up to 3900% cheaper vs SQLServer on Azure).

[1] https://docs.servicestack.net/ormlite/litestream

3900% cheaper makes no sense.

I like that "connection pooling" is just limiting the number of open handles in a LRU cache. It's also interesting because instead of having to manage concurrency at the connection level, it handles it at the tenancy level since each DB connection is single-threaded. You could build up per-DB rate limiting on top of this pretty easily to prevent abuse by a given user.

Is there a straightforward way to set up Litestream to handle any arbitrary number of DBs?

The v1 backend system was optimized for rapid development and served us well. The v2 backend will be somewhat less flexible (no joins!) but is designed for much higher scale.

This way it's possible to get all network data from a single place (the BGS) rather than having to connect to every PDS, which is simpler for consumers and dramatically reduces the workload of PDS hosts.

Some details about event streams here, although the APIs are still evolving: https://atproto.com/specs/event-stream

"Big-world" networking by Big Tech-to-be Bluesky with super-powers, I wonder? Is this BGS also going to be federated, or is that the big centralized beating heart of this platform managed exclusively by BS?

To me, on the surface, particularly assuming you are building a distributed system to be run and deployed by many users, some of which are not professional sysadmins (which I believe is likely to be a goal here, and should be), this seems like quite a sane choice. I'd definitely expect a design goal to be avoiding the need to setup/configure/look after any additional database or other servers.

Simplification of installation seems not like a good enough reason to trust your whole backend on this. Installing and maintaining a database-server is not that hard today. This is well established and documented, unlike this. But I also don't know enough about this app, maybe this is just one of several options, meant for a specific usecase? Using this in a standalone desktop-app would make sense, while still offering a mature sql-backend available for server-installations.

SQLite is just about as mature and well-tested as it gets in the entire world of software: https://www.sqlite.org/testing.html

Each users' data is naturally partitioned at the atproto repository level, so this is the sweet spot for per-user SQLite databases. It would make total sense for a PDS instance to have just a single user on it, and in fact that is likely for many self-hosters. It's also worth noting that the PDS software already had SQLite support, which made this change somewhat easier.

There are legitimte trade-offs to this kind of a system but it comes out way ahead in this case, and it's not as wild as it may seem to those not familiar with the power of SQLite.

A major consideration is that we're planning to run at least 100+ instances, which would require operating 100+ high availability (primary+replica) Postgres clusters. This would be a huge amount of operational and financial overhead.

We chose this route because it is better for us as a small team, with relatively limited resources. But it also has the property of being much easier for self-hosters, of which we hope there will be many.

[..] Each user has their own SQLite file [..]

[..] We also introduce 3 separate SQLite databases for managing service state [..]

This doesn't use SQLite for the database-managment, but for the individual "document". The database-managment itself is handled in the application-server. You jiggle around files and poke wherever it matches, this is by a classical filebased database-managment-system.

> It would make total sense for a PDS instance to have a single user, and in fact that is likely for many self-hosters.

Sure, if it's just a low-user-instance, the performance is not much of a deal. But from my impression here, this is also the code Bluesky uses for everything else, from low to massively high user-instances. And then I want to see how RAM holds up, when you have 10k+ user-databases open at the same time on one instance.

> There are trade-offs to this kind of a system but it comes out way ahead in this case.

Which is why I want to see some actual numbers and solid explanations going more into details then the gossip in the comments here.

> A major consideration is that we're planning to run at least 100+ instances, which would require operating 100+ high availability (primary+replica) Postgres clusters.

Are those independent instances, or just 100+ instances servers from the same company on different locations? But I don't see how this can replace a whole postgres-cluster without removing significant functionality. I mean sqlite does not have good replication on it's own AFAIK, so as you seem to still use replication, you just replace it with another solution? Which also means you remove the same options for anyone else, and forces them to use your solutions?. I don't see how this will be beneficial for self-hosters.

Think email. When you send an email and CC five other people as well then seven people now have the same copy of the email stored on their email servers. That is, there’s no central database that contains a single email that is referenced by others.

This is basically how sharding with relational DBs works as well.

This sort of data denormalization is almost a requirement as applications scale and especially for many-to-many applications that have a high write to read ratio.

Low write to read and you can get away with a single master to many slave relational DB architecture for quite astonishing numbers of requests and data!

*(actually they just onboarded a second production PDS yesterday.. progress!)

Haven't done any research to determine if there are plans for direct messages.

Security is not a concern here. It's just literally bucketing ids. Also, this is not needed with modern file systems.

Or, like me, they're drowning in security tooling from corporate and don't want to have to carve out exceptions for md5 usage in each.

With their scheme, collisions are already guaranteed to happen if they have >256 users.

They said:

if you need security don't use md5.

If you don't need security, use something faster than md5.

md5 is neither secure nor fast, why use it at all?

There's a dedicated waitlist for developers that will get you access quite quickly: https://atproto.com/blog/call-for-developers

As a comparison, Cohost limited account setup when it launched as a way to limit growth. But it didn't lock viewing the entire site behind an account requirement because... come on. What does that have to do with scaling, we all know why that restriction is there :)

To be fair, it seems to be working. Needing to seek out and find invite codes means that signups are more visible -- signup codes get shared over social media and that means mentioning Bluesky publicly and keeping it in people's minds. It also forces people to ask publicly about access, which makes the network feel more exclusive and turns every signup or expression of interest into an advertisement for the network. It's a good marketing strategy, and I suspect that a nontrivial portion of Bluesky's current buzz comes from that marketing strategy, so I can understand why it hasn't been abandoned yet. I mean, look at the current thread; if people didn't need to coordinate publicly on HN to get access then this subthread wouldn't exist and then there wouldn't be a public thread where a bunch of people express interest in trying out the network -- and that publicly expressed interest in this very subthread makes Bluesky feel more in-demand.

In fact, this is such an effective marketing strategy that I've seen Bluesky users complain that invite codes are too common now and that their invite codes aren't in as much demand as they used to be. That FOMO loop is so powerful that it's even affecting the people who already have access to the network who enjoyed the feeling of being in control of an artificially scarce resource.

But sure, all of this is definitely not a growth hack, I believe you ;)

Regardless of whether it's good marketing, the account requirements make the platform a lot less relevant in any serious discussions about the direction of social media, because despite its plans for the future for federation and access, what Bluesky is today is a platform that is in practice even more locked down than Twitter is.

Me: "if the network is intended to be public, why are user profiles and posts currently hidden behind a login wall?"

Paul Frazee: "it was a kind of bad artifact of how we set things up initially (just trying to ship). once we realized it communicated the wrong idea it was too late, and we now need to spend a heavy bit of effort communicating before we spring it on everybody."

> Once we realized it communicated the wrong idea it was too late

For what? Too late to change the technical side of things? Is there a major technical barrier to having a public interface that matches the public firehose APIs? Because I can't figure out what that barrier would be.

What magical deadline or restriction was in place that would have prevented fixing an obvious barrier to the network?

And "once we realized it communicated the wrong idea"? People aren't misinterpreting the message, they're accurately assessing that Bluesky is not an Open network even though it is marketed as one and pretends to be one. This isn't a communication problem, it's not that blocking public access communicates the wrong idea to the public, it communicates correctly that the network isn't Open. It's complete nonsense to try and phrase a failure to fulfill the basic promises of the network as if it's actually just a PR problem.

----

> and we now need to spend a heavy bit of communicating before we spring it on everybody.

Communicating to whom? The users? Is this an admission that Bluesky users don't view the network as public or that they don't want the network to be public?

This is phrased like "we need to spend a bunch of time clarifying and explaining how this will all work before we pull the rug out from under people's feet" but who on earth would this be pulling the rug out from under? Who would be confused about this change? This isn't actually complicated; if a user without an account looks at a post it will either be visible or it won't be visible. That doesn't require a FAQ.

If the idea of that post being visible is contrary to community expectations and if the devs feel they literally can't make open decisions because the community would oppose those changes, then that's a pretty heckin big problem and it sounds like they should stop advertising that this is intended to be an open network or that federation is coming any day now, because it doesn't sound like the community is on board with that idea.

Or is it a communication problem for people outside the network? But how? What would that even mean?

Who outside of Bluesky would be confused if the devs took measures to fulfill the promises they've been publicly making since day 1 of the network? This isn't some complicated thing where people will be misinformed or they'll be confused by the idea that they can view without an account but can't post without one. That's how most networks work, locking viewing behind a login is the abnormal confusing decision to outsiders.

Ultimately, the network will be publicly viewable or it won't be. I do not understand what about that would require a PR campaign. Were people signing up for Bluesky thinking that the network was going to be permanently private? Because if so, that is something the owners should be horribly embarrassed about.

----

It's just a fundamentally weird statement. The only thing that a closed-down network communicates is that it's closed down and exclusive. The only reason that fixing the network to reflect their own marketing would be a problem is if the network doesn't want to reflect the marketing. In which case, they should stop pretending that this is a temporary limit on growth to help prevent out-of-control scaling.

The most charitable take I can have about the response is that it's corporate bullcrap from people trying to take a simple decision that was made for marketing reasons (or has accidentally been found to be extremely valuable for marketing) and to after-the-fact justify it as something complicated and difficult so that they have an excuse to avoid actually making the change.

The less charitable take I could have is that they're being honest, and they're unable to make changes to make the network more open because their userbase would be hostile to those changes or without a PR campaign would view opening up the network as if it was an attack -- and if that's the case, that sure as heck is not making me feel confident that this network has any potential at all as an Open platform. If the users aren't on board with Bluesky as a federated and Open network for everyone, then y'all don't have an Open network and it doesn't matter what your plans are.

And in either case, it's clearly not a temporary technical restriction to help with scaling so I don't know why devs are jumping into threads pretending that it is. It's clearly a deeper problem or else the devs wouldn't be giving you this kind of a nonsense response as soon as you tried to dig into it more.

Edit: they're all gone!

Thanks.

My mail is at the bottom of my bio.

EDIT: I'm fresh out for now, sorry!

But yes, anyone is free to operate a BGS. It does necessarily require a non-trivial amount of storage, compute, and bandwidth. A funded startup, well-funded non-profit, or any just about any cloud provider could likely afford to run one.

It's also entirely possible to operate a BGS that only mirrors a slice of the network (for instance, only users in one country) if desired, which could in some cases make it affordable for a single user or small coop to operate.

In practice no one will switch because it makes no sense to do it. If there happen to ever be more than one real BGS contender, it will be from something like Cloudflare that will just replicate everything Bluesky Inc decides.

I‘d really love to have some more civilized hub again that isn’t full of hate and anti-intellectualism.

It is a shame, as it seems like a nice alternative that has some cool ideas.

Props to whoever actually reviewed that, you are a warrior

1. require reviews before merging

2. have not very disjunct PRs (sometimes for e.g. legacy maintenance projects you mostly have disjunct PRs normaly you do not)

then you need stacked PRs for productivity, i.e. you need to be able to continue working on a new PR based on the old PR before that is fully merged (or reviwed).

In this case in my experience three workflow work:

1. you (may) squash commits, and rebase stacked PRs once the previous PR has been merged (or sometimes majorly modified, but that quite advanced rebase usage). This works but has some major pain points: 1) rebased during reviews are terrible bad handled by github, 2) git doesn't keep track of the original start of a branch, this can lead to issues if you squash the commits when merging, 3) no good build in tooling for it

2. All forms of history manipulation are forbidden including rebasing and squashing. It's merge only because of this git doesn't get confused when merging squashed commits and everything seems fine... Until you now realize that follow up changes from reviews of a parent PR happen in the git history chronologically after your follow up PR and that can be a total pain depending on what changes. (Through you are allowed to fully rebase your history before marking a PR as ready for review so as long as the "stack" of PRs isn't too deep it's fine).

3. you agree with Linus that github PRs have major issues and go with a patch based approach for merging, now you need completely different tooling which often less nice modern UI but id doesn't have any of the issues of point 1 or 2

It's was quite a wtf are you doing industry moment when I realized that the most widely used contributions flows (weather in open source or in companies) are either quite flawed (1&2), productivity nightmares (no stacked commits) or quite inconvenient (3).

I almost never look at them, but once in a while it is really great to see the thought process that led to something.

* Refactor function `foo` to accept a second parameter

* Add function `bar`

* Use `bar` and `foo` in component `Baz` to implement feature #X

If you give me a commit history like this, I can easily validate that each step in your claimed process does what you describe.

If you instead give me a messy history and ask me to read the diff, you might know that the change to file `Something.ts` on line 125 was conceptually part of the refactor to `foo`, but I'll have to piece that together myself. It's not obvious to the person who didn't write the code what the purpose of any given change was supposed to be.

This isn't a huge deal if your team's process is such that each step above is a PR on its own, but if your PRs are at the coarseness of a full feature, it's helpful to break down the sub-steps into smaller (but sane and readable) diffs.

That said, OP is in an environment where it sounds like this kind of structure is already the cultural norm.

Thanks!

What you should be doing is breaking down PRs more finely so that your unrelated refactors are all separate single-commit PRs. That ofc requires that your pr review round trip time is fast

You could have simply randomized the text in each commit, put the ticket id and the one "why" in the merge commit body and gotten the same end result amount of real information in the end.

* Priming the reader so they are able to quickly interpret what they're seeing when they open the commit.

* Making it easy to search or scan for a specific change.

The last commit message in my example would probably have included the name of the feature as well as the ticket number, but I couldn't be bothered to invent an actual feature name.

DRY doesn't really apply to technical writing, at least not as extremely as you seem to think it should. Headings are supposed to summarize the contents, and that's what commit messages are: headings.

I'm trying to imagine the near infinite terms I would have to search for to find the commit where I "changed from a hash to a set".

Regardless, every other thing you said could also just be done in the central PR body (and thus the merge commit) and be much easier to access.

Instead of "priming the reader" it's infinitely more helpful to tell the reader why you did something, because you can't extract that from a diff.

Again, that can go in the PR body or in subsequent lines. You have ~50 characters in that first line, which is never going to be enough to fully explain anything.

I'm also not suggesting that you eliminate the PR body: that should also include more context. All I'm suggesting is that taking the trouble to organize your commits into discrete units helps reviewers to understand how you perceive the various changes in a single PR as being related to one another, and no amount of text in the PR body will provide the same benefit as being able to look at several distinct diffs containing related changes.

loop i up to n times

break when false

check value returned is not null

I.e. with an idea like:

- if we try to commit so that each commit does a singular change

- then by limiting the number of commits we limit the number of "logical" changes in a PR

- and in turn make reviews and similar easier

1. Branch f-prime from master. 2. Squash merge f to f-prime. 3. Pull request f-prime to master. 4. Profit.

Someone reading the git changelog 5 years down the line most likely wouldn't be able to find your "novel" in the PR and definitely won't appreciate if instead of a "novel" you ended up with a "short call" with the assigned reviewer explained what you actually did in your 50 "wip" commits.

Anthing else, I call it a landfill site, not a maintained repository.

In fact, I'd go as far as using their commit habit as a measure of a candidate's consideration for their colleagues.

Diffs are great but sometimes they're just as overwhelming in a huge PR. It's nice to first follow 5-10 commits in chunks of logical change.

I suspect squashers use the wrong tools. Use source tree, or, if you are on linux, smartgit. You can see a detailed log, which makes it much easier.

But then when you're done, turn it into a series of patches for a reviewer to read. In the words of Greg Kroah-Hartman, "pretend I'm your math teacher and show your working".

In a maths assignment, you spend ages making a big mess on a scrap of paper. Then when you've got the solution, you list the steps nice and clearly for the teacher as if you got it right first time. In software development, if you're not a dick, you do the same. You make a big old mess with loads of commits, then when you're done and it's review time, you turn it into a series of tidy commits that are easy for someone to review one-by-one.

Am I living in a bubble and all the glorified 500k TC FAANG devs from HN really routinely submit a changes consisting of a tangled mess of 50 "wip" commits for their code review without any repercussions?

If your PRs are tiny it's not a big deal, but with 190 files changed in this one, it absolutely should have been rebased into a more reasonable commit history.

I don’t want to interrupt my flow with intermediary commits.

With commits like "typo", you might as well squash these into the commit which introduced the typo in the changeset.

If there are changes across many files, and the changes were made automatically with some search-and-replace (or some refactoring tool).. by having a commit that's only that automatic change, it's easy to look at that commit and tell what the changes were. -- Presumably, non-automatic changes are going to be smaller.

I guess roughly, if it makes sense to apply a changeset that changes 5 things, you'd want 5 commits. Having commits like "typo" means there are more commits; but squashing those 5 things together makes it harder to discern the granular change.

Or a ghost.

Bluesky has a main PDS instance at https://bsky.social that serves almost all of the Bluesky user base.

There is a good overview of the architecture here:

https://blueskyweb.xyz/blog/5-5-2023-federation-architecture

Here’s a snippet from the protocol roadmap they published 3-4 weeks ago [1]:

Multiple PDS instances

The Bluesky PDS (bsky.social) is currently a monolithic PostgreSQL database with over a million hosted repositories. We will be splitting accounts across multiple instances, using the protocol itself to help with scaling.

[1] https://atproto.com/blog/2023-protocol-roadmap

The main motivation in moving from a big central Postgres cluster to single tenant SQLite databases is to make hosting users much more efficient, inexpensive, and operationally simpler.

But it's also part of the plan to run regional PDS hosts near users, increasing performance by decreasing end-to-end latency.

The most experimental part of this setup is using Litestream to replicate these many SQLite databases (there are almost 2 million user repositories) to cloud storage. But we're not relying on this alone, we're also going to maintain standard SQLite ".backup" snapshots as well.

  bsky-social-scbch-eolha
  bsky-social-fs26y-d6gnv
  bsky-social-2lx5u-ntrdv
  bsky-social-hboq7-dyuue
  bsky-social-b2v3f-3a23q

    bsky-social-lkzsp-7x7ja
    bsky-social-p4vwr-nrthu
    bsky-social-bdu6c-6tbv4
    bsky-social-fkpgk-oestw

How many of them active?

These kind of movements makes me think they're not serious about scaling up. Wouldn't surprise me if then end up as an also-ran

The exceptions are those rare products that despite a low cost marketing sell themselves so well that their organic growth is fast and in a few months everybody use them.

Maybe Bluesky don't have the money to advertise or is not compelling enough. As one data point: I know about Mastodon but I think that I learned about Bluesky only today. I went to their site and there is nothing to explain how it works except that it's some social thing. I learned more by reading the comments here. Apparently it's being marketed at a very low cost.

Edit: I'm all out now :)

But the interesting thing for me isn't activity — it's the people on there.

Of the cohort who had >100k followers on Twitter, I think more of them post regularly on Bluesky than post on Mastodon. Bluesky definitely has a more cohesive feel, especially because there's currently just one instance & mod team.

I wonder if BlueSky intends to follow on those. For example, hiding user actions counts (repeats, favourites, etc...) until the user acts on one.

Things like these may be strange for those accustumed to Twitter, but personally, is what makes me stick with smaller instances on the Fediverse.

To me, the ActivityPub network (Mastodon and friends) is relatively unique in the social media space in having no direct commercial pressures (the protocol is developed by W3C) and therefore being inoculated against the causes for these dark patterns.

https://vqv.app/stats/chart is useful for Bluesky and draws on the Bluesky firehose for data. https://stats.nostr.band/ seems useful for nostr.

It's pretty active during North American hours.

bsky-social-etdu7-njigu

bsky-social-2ktcs-uwoxg

bsky-social-6f5nh-36gnq

bsky-social-ciwro-3gzk5

bsky-social-y4h57-dxh3g

E: Happy to take one, if somebody happens to have a spare one left. Email is in my bio.

bsky-social-h3d4w-u6yn4

bsky-social-74bqi-vkmcq

bsky-social-n3fdq-46nxz

bsky-social-yippe-32vdr

bsky-social-l2fbt-xnscx

bsky-social-ge2mz-mfmpi

bsky-social-hykwa-x3ox4

bsky-social-gh4mt-2od6p

bsky-social-dejzy-mmcxf

edit: all gone :(

7poji-p36pm

irn4h-ncvic

2hb2e-xhxnb

2k4na-5qiqu

bsky-social-zigwm-f3qpq

bsky-social-2jlu7-apy5a

bsky-social-6ct52-4egmz

bsky-social-cy64m-53sqn

e: second one now used, first still up

e: both used

e4: check out dns on [my username].com

Why not use Postgres with RBAC (Row Based Access Control).

- simpler cloud architecture

- simpler resource management

- simpler partial backups/restore

- simpler compliance with law enforcement

- partitioning might be easier, e.g. when handing "user account storage which should be undo-able for a while" (e.g. long term absent users data could be moved to cold storage, blocked/deleted users data could move to some scheduled for deletion space allowing undoing it for a while but then reliable auto deleting them, a copy of users data where crime detection triggered (e.g. CASM) could be moved to a quarantine space, etc. And each of the spaces can be completely different servers with different storage methods and retention policies, virtual access control and physical access control. Sure you can have all of that with RBAC + partitioning + triggers + roles in postgres, but it's the personal data store of a user so you don't need cross users FK constraint enforcement and it makes it much easier to make sure you don't miss anything wrt. access controll or forgetting to partition/move some columns of a new table etc.)

- maybe simpler billing for storage ("just" size of DB)

now simpler doesn't mean better, but often it pays of as long as you don't run into the limits of what is possible with the simpler architecture (and as far as I can tell you can shard this approach really nice, so there at least there shouldn't be scaling performance limits, scaling cost and future feature complexity limits might still apply)