He’s a seasoned database architect. With SQLsync he’s made a way for frontend developers to query and update a remote database as if it was completely located right in the browser. Because it basically is. The power of WASM makes it possible to ship a whole SQLite database to the browser. The magic is in how it syncs from multiple clients with a clever but simple reactive algorithm.

It’s a radical attack on the whole problem. Much of our work as developers is about syncing data. When you start looking at React and REST APIs as a kind of sync procedure, this approach can open a lot of new possibilities. You don’t have to write a weird bespoke database of trees of objects fetched and cached from the API any more. You can just update and query it locally, with all the power of a relational database.

It's hard to adopt such a system in most traditional web shops with their specialized backend and frontend teams. You're pulling out the database, backend, transport, and auth layers and replacing them with this single block system. Most system architects grew up in the backend so they are generally pretty ignorant of this issue. As it touches both sides extensively you're probably not fitting this into an existing system, which leaves only green field new development. Finally your backend is not an AWS or Asure service, neither is it lambda friendly. All of this means that most architect types I talk to will never touch it.

This style of system mostly already exists with old tech, CouchDB+PouchDB. Which works pretty well for some things. The downsides are that the query system isn't really ideal and the auth and data scoping system is pretty foreign to most people. The easiest model to work with is when the data is totally owned by a single user, and then you use the out-of-the-box database-per-user model. High data segmentation with CRDTs removes a lot of conflict issues.

It has scaling issues though, CouchDB has really high CPU requirements when you're connecting 10k to 100k users. The tech is long in the tooth though it is maintained. On the system design side it gets really complicated when you start sharing data between users, which makes it rather unsuitable as you're just moving the complexity rather than solving it.

This approach seems to hit the same target though will likely have similar scaling issues.

Look forward to see the evolution of the system. Looks like a first step into the world.

I remember having problems with 200 users serving from my above average dev pc for testing internal tools.

It's a DX dream though.

A "native" electron or capacitor distribution system is a fine extension of a local-first web client. And an advantage of building fat clients generally is they lend themselves to such distribution models much easier than say, htmx or hotwire.

Native fat client have had their benefits and lots of people still prefer them, but always had the drawback of manual data management and installs. Being able to leverage any device you own with a cloud synced local-first client really gives you the best of both worlds.

But not all software fits readily within this model.

First, my list failed to include web because of the context. Web is, by far, the largest and most important platform. Even if I'm building only native installers for a project, I need to be able to build web projects with the same tools to make this work.

Java also fails "one code base" requirement as desktop and mobile are very different. The poor iOS support is going to cause it to fail the "square on" requirement as well.

No on Java.

If economics don't line up then you have to pull the plug and set them adrift, which is much easier and more secure with a fat client that runs without a server than say, a complex cloud system.

This sounds like the set up for a "No? Me neither." punchline. Certainly one of the features of Flash is that it gave you fairly good consistency across computers, but honestly my perception of Flash wasn't that it was going to solve some grand web problem, but more "oooh look, shiny rollover buttons!" and "ooh look, super obnoxious advertisement that forces people to pay attention to it!"

That's the interactivity aspect. At the time, it was the bees knees of UI capabilities.

You create a database per user and the auth and sync are all self contained within that database. This system is multi-master, which means that any change on a client or on the server will be replicated to every other. There is no "authority" which trumps the others. The server is simply a central hub that requires the right authentication to allow the sync process to happen.

When you want to create a set of data that crosses user boundaries, it gets complicated. It's possible to do, but you're not on the easy train anymore.

Creating a system that's both easy to use, and scopes the right data view out of the system wide tables and rows we usually think of databases, is not the CouchDB nor SQLSync model.

So it sounds like this excludes most enterprise use cases?

[1] https://fireproof.storage/

First to address the main point: why not cache the relevant bits in some kind of local storage. SQLSync plans on doing this, specifically using OPFS for performance (but will have fallbacks to localstorage if needed).

Second to address the question of why not use built in kv stores or browser side databases. One answer is another question: how do you solve sync?

One approach is using a data model that encodes conflict handling directly, like CRDTs. This approach is easier to put into general kv stores, as syncing requires simply exchanging messages in any order. I find this solution is well suited to unstructured collaboration like text editing, but makes it harder to coordinate centralised changes to the data. Centralised changes are nice when you start introducing authentication, compaction, and upgrades.

Another approach is doing something similar to how Git Rebase works. The idea is to let the application state and server state diverge, and then provide an efficient means for the app to periodically reset to the latest server state and replay any unacked mutations. This approach requires the ability to re-run mutations efficiently as well as efficiently track multiple diverging versions of the database state. It's certainly possible to build this model on top of local storage.

For SQLSync, I found that by controlling the entirety of SQLite and the underlying storage layer I was able to create a solution that works across platforms and offers a fairly consistent performance profile. The same solution runs in native apps, browser sessions (main thread or workers), and on serverless platforms. One of my goals is to follow the lead of SQLite and keep my solution fairly agnostic to the platform (while providing the requisite hooks for things like durable storage).

https://en.wikipedia.org/wiki/Web_SQL_Database

After 10 years of building SPA "web apps", that data synchronization mechanism feels ahead of its time.

Ultimately that comes down to whether you can implement a process to ensure consistent resolution of any incompatibilities between multiple concurrent updates. Sometimes that can be done, and sometimes it can't, and which is the case for your application depends on your business rules, not on any technical capability.

If your business rules don't allow you to implement a resolution mechanism, you need locking so that only one update can happen at a time, whether you have the technical capability to support concurrent updates or not.

Every hard problem I have today in my career involves getting business people to define their business problem properly in order to solve it with technology. Even the hardest code I've ever written was easy compared to some projects, simply due to the business issues lurking around the project. Last week I finished a script to download a CSV and save it to a SQL table (literally) that took 3 weeks because business folks couldn't get their act together on what they wanted. I finished another project in a few days which is currently the core of a previous employers energy efficiency controls product which was easy because the person defining it did it very well, and I had no questions, just work to perform.

Traditional ACID systems can't really handle them nicely - your only choice with an update is to commit it or discard it - so you have to do a lot of handwritten logic on top, and even if the database itself handles that well, the layers above it generally don't. Event-sourcing style systems work well but they're still not really mainstream yet.

But just like mentioned it is hard to convince people that it is what they actually want.

People fall into some grand illusion that everything should be always available but in reality then one person is doing changes at a time and if somehow 2 or more people have to work on something - more often than not they should be talking or communicating with each other anyway to synchronize.

Even with GIT and fully distributed development you cannot solve conflicts automagically. You still have to communicate with others and understand context to pick correct changes.

At some point we have to communicate which change is correct.

It does not have to be synchronous and it might be via commit message - but still change alone is not enough for conflict resolution.

If you edit word document and someone then changes something there is no commit message but might be comment on document, email or im.

For some things, you need the time tested solutions.

[1] https://en.wikipedia.org/wiki/Revision_Control_System [2] https://en.wikipedia.org/wiki/Concurrent_Versions_System

So many forgotten locks from lazy devs...

[0] https://en.m.wikipedia.org/wiki/Azure_DevOps_Server#TFVC

So long ago I can't remember exactly which but I was running a local cvs and/or subversion repository for my own work just to avoid issues like the above. s [0] https://blog.codinghorror.com/source-control-anything-but-so...

[1] https://developsense.com/visual-sourcesafe-version-control-u...

To get back on topic, the key thing an explicit database gives you is a purpose built-language (and data-integrity enforcement etc. if you do it properly), that everyone knows. (Or used to? SQL is getting more hidden by abstraction layers/eco-systems these days). I'm old, so I reach for my older, well understood tools over new and exciting. Get off my lawn. It may be over-architecting, but I'm also not working in maximising 'performance in milli/micro-seconds is vital' high load environments, or releasing updated software every other day.

The other issue is tool/eco-system fragmentation.

But when you're young and have the energy and mental capacity to abstract out the wahoo for effeciency/performance, you do, because you can, because its better at the time. In our day everyone was writing code to write to code which were effectively the pre-cursors to ORM's. It's just part of being young and committed to your craft, and wanting to get better at it - this is a good thing!

It's only as you get older you start to appreciate the "Less is More" around same time that job ads appear with "Must have 3 years of SQL-Sync experience" (no offence intended here). There are both costs and benefits but which and how much of each you only find out years later.

Now, the best part of this shit show was they had ~30 different customers and each of these customers had a clone of the main thing that would be customized. So the “TFS Guy” had to determine if to keep in the customer clone only or to propagate to the main and then to all the other clones!

Needless to say the “TFS Guy” made a lot of money.

[0] https://jedivcs.sourceforge.net/

The problem with databases is actually complexity. Any individual feature is more or less safe, but around the time reliability, caching and indexes get matched together there is a complexity explosion and it doesn't (normally, anyhow) make sense to implement a domain-specific DB (call is a DSD?).

But, around the time a company has invested in implementing those 3 features and discovered that it has sunk a lot of resources into the DSD, is politically averse to recommending it be stripped out and there is a high real cost to taking out the tech debt in one go.

Really the problem here is SQL's syntax. If using a basic relational database was a pleasant experience that involved some familiar C-like syntax instead of broken English people would be more tempted to go with a DB instead of rolling their own. The NoSQL databases were a good step in that direction, but then they by and large overfocused on big data instead of everyday usefulness. Things like Redis took hold which is nice.

Making it easy to run SQL is a reasonable approach, but the problem is that the good databases - I like postgres - are SQL native and it is hard to get efficiency without speaking the DB's language. We really need a PostgresPostSQL database that is a perfect postgres clone but primary parser supports a language with good syntax.

- No non-nullable types (at the expression level). No way to express e.g. normal boolean logic

- No real data structures (at the expression level), unless you count rows, which are not first-class values. Even collections aren't first-class

- Very awkward control flow constructs. E.g. look at how you write recursive queries. Even if/else is weird. It's a classical Turing Tarpit: everything is possible but nothing is easy

- Absurdly complex grammar. Something like 200+ keywords. Bizarre and inconsistent quoting rules

- Weird rules about where expressions can be used, and where you have to put variables. Often if you want to extract out a common subexpression and reuse it you have to completely restructure your statement.

- Missing basic functionality, e.g. no string concatenation operator (except in some nonstandard dialects)

- No libraries. No standard way of reusing code. No unit testing framework, which means in practice most projects have no testing.

It's a terrible language by any reasonable standard. Its closest cousin is COBOL, and that's honestly less awful to write.

That's no excuse for making it bad.

> No Libraries? There are a zillion battle tested libs out there.

Not in SQL.

> And why you would unit test datasets in a database?

For the same reason as anywhere else.

> Most if not all software stacks have mock libs available.

Exactly. They're a basic piece of functionality you'd expect most languages to provide, but not SQL, because it's awful.

`||` is the SQL standard. IMO its biggest deficiency is a single NULL nullifies the whole thing. But at least that's consistent with how NULL behaves in many other cases.

We have dozens of programming languages in use for general programming, it is a field that is constantly evolving. Even JS, which is hard to change because that's what browsers run and you don't control your client's browser is seeing some evolution, using transpilers, and now WebAssembly.

But for databases, there is only one, and that SQL. There are some alternatives, but no one comes close to SQL in terms of usage. So maybe SQL is not that bad after all.

Maybe the reason is that the relational model is really good, and attempts to deviate from this will only work in a niche. That a declarative style is also really good, and again, you won't have much success if you deviate from this. And if you end up doing SQL with a different syntax, for most people, it will be too small of an improvement to justify changing their way.

I'm very happy to see the discussion thus far. This first post focused on the parts of frontend engineering that motivated me to create SQLSync rather than specifically how SQLSync works. I'll be addressing that in my next post. Thanks for all the great questions! Keep them coming :)

My only saving grace is that technically I didn't make a new DB - just using good ol SQLite. Mostly. :)

I fear sync'ing databases instead of client server models is one of those - either your sync mechanism will just melt, or there are deep assumptions not met

Inwoukd feel safer building a set of CRDT primitives to work with if I feel the need for fast UI and stick with forms submit for everything else -

The underlying issue is that users of SQLsync are going to assume that consistency works, but in suble and unsuspecting ways it won't.

As far as I can tell, the only solutions that handle distributed consistency are those that use CRDTs.

Also, I'm considering adding a CRDT column type to SQLite to enable embedded collaborative use cases while using the relational model for the rest of your data's structure.

You can sidestep it altogether if you make mild UX sacrifices and revert to something more akin to the PHP/SSR model. SPA is nice, but multipart form posts still work. Just the tiniest amount of javascript can smooth out most of the remaining rough edges.

Our latest web products utilize the following client-side state:

  3rd party IdP claims for auth
  1st party session id in query args
  The current document

Optimistic rendering means your frontend and backend are tightly coupled, error recovery and synchronization is much more complex, and you are locked into (likely heavy) frontend rendering patterns that add even more complexity and coupling.

We've spent well over a decade trying to avoid the fact that frontend actions require backend logic to complete. Its a losing battle that's just made worse by trying to paper over it.

Edit to clarify: this isn't a direct comment on the OP tool. I haven't used this tool directly but having a hunch it does solve some important use cases for common patterns.

at that point there's already a diverging state problem, even if it's saved to localStorage (but obviously it's much better to save it on the server)

it's absolutely ridiculous that we spent the last 10+ years simply reinventing the same wheel (all low-level tools, Angular, React, Vue, rxjs, hooks, signals, whatever, etc), without providing the useful composable primitives for the hard problems.

and now there's yet one more thing on the frontend, SQLite. okay, there's a sync thing too. it's not like we didn't try it with that adorable DB (RethinkDB).

Can't say I've been asked to build optimistic updates for that reason, but the irony there is pretty rich if that's the UX goal you've seen it used for.

In my experience optimistic rendering actually creates more data loss risk than it solves. Caching layers are involved, state is duplicated, and routing comes into question as users could leave the page before a failure state is returned.

For example the user wants to interact with the service, order something, check for new messages or send one.

On the happy path things are trivial, easy, synchronous, even quite resilient. There's not much complexity to manage on "no new message" or "add to cart, yes, buy with saved card, ship to saved address, k thx bye", if the browser or server crashes, or there's a network error, the interaction is short, easy to retry, problems are not hidden (quasi instant feedback)

But as the total interaction time increases, as the state the user would need to reproduce grows (adding new card, shipping/billing address, adding a lot of stuff into the card, writing a long message) the need for partial state sync arises.

See: https://news.ycombinator.com/item?id=37584049

SQLSync also looks very interesting. I'll try it out in my next project!

[0]: https://tanstack.com/query/latest/

Tons of work to do in order to really prove it out though. But I'm pretty excited to keep pushing and see where this goes.

Using solutions from the Hotwire or htmx family would mean that a query is just a server query - making those fast is a better-understood problem.

Yes, they should, because I _need_ good feedback for connection issues.

I have not used those two specific apps, but for other "online first" apps it's such a common problem. Open an app, type a note, switch back to a different app (or turn off your phone, or close laptop lid).

Later on, you want to access the note and it's not there. Why? the authors decided to be "smart" and "offline first" and made sync in background, with no feedback, or with heavily delayed feedback... and I was walking out of wifi range when i was typing the note, so connection was spotty.

The OP's demo TODO app has exactly the same problem - no indication when the data is already uploaded. So please, if your real goal is collaboration, let user know when you cannot perform it, don't sweep it under the rug hoping the things would get uploaded eventually.

In regard to the TODO demo - I should totally add that. Thanks :) https://github.com/orbitinghail/sqlsync/issues/31

I think my sibling comments are completely underestimating the complexity and the market value here. Apple Reminders or Google Tasks are customer facing applications for single individuals, nobody but a single client is working at a time on their own data. Businesses on the other hand want their data at a central location and personnel close to the tasks using client applications. They don't want any reliability issues in the client-server-communication blocking the work being done. Heck, businesses want to be able to throw more people at the same tasks and we call it collaborative editing.

That would allow offline viewing to a variety of apps, regardless of approach.

Last time I checked Google Docs doesn't primarily allow editing offline files, which hints how hard it is to support substantial features beyond mere reading.

I just don't think there are "embarrassingly client-side, but also needs a server for some reason" web apps that would benefit from this in the real world. Even Google's or Apple's version of the simple Todo app has a lot of (useful) integrations, which means having some level of trust with the client.

The marketing materials for htmx are also a bit off-putting to me, and the authors seem to be enjoying fueling the flames of old and useless holy wars about what REST and hypermedia "actually" are, and how we all need to get back to basics, and remember to thank Ted Nelson before every meal, etc. Their online book spends something like 30 pages explaining what hypermedia is... [2]. I prefer not to sign up for a religion when I choose a JS framework (honestly, that's a bit hard these days :-/).

--

1: https://github.com/bigskysoftware/htmx/blob/master/dist/htmx...

2: https://hypermedia.systems/hypermedia-reintroduction/

Yeah, htmx is 3800 LoC, and you could do an 80/20 version of it for a lot less, but there are a lot of details to get right: history, collecting inputs, etc. plus, since it's a general purpose library, it has an extensive event system, extension mechanism, etc. There isn't a ton of dead weight to drop, although 2.0 should be smaller as we drop some bad ideas in 1.x.

I don't care too much about the holy wars around REST, I found them off-putting back in the day before I really understood the concept, but I am passionate about the uniform interface and communicating to developers why that was interesting and different. I do go a bit hot at times, especially at twitter, but, on the other hand, if I didn't, would you have ever heard of htmx? I try to balance that all out w/ reasonable essays and the book, which, I think, is worthwhile for most web developers to read (it's free online.)

I personally prefer InertiaJs [1], which is some kind of front-end router system with its state synced with the server in an "old style" fashion.

[1] https://inertiajs.com

I too, enjoy satire.

This especially becomes true for products that are expected to work in low internet zones.

“Just don’t run services that require more than one VM.”

It’s a ridiculous take.

SPAs are just a subset of all the JS we can possibly write - interactivity is not compromised.

On iOS/Android, before all the WebKit apps took over, many apps would use a local SQLite database to support offline edits and implement a syncing protocol to the server. It's a lot of work, but the end product can be quite handy. (This is how you'd expect your email client to work, for example.)

An average SPA is, in the end, a website for most users. Take a payment form for instance - we had those 10, 20, 30 years ago, with 0.1x the effort.

Of course there's place for fat client tech, but there are 1000 websites for each 'email client' - exceptional cases often dominate the discussion.

Running server-side does seem to be one of the problems SQLSync wants to handle? I wonder how well it does at that compared to other ways of doing it?

One can solve those without reinventing the database, as the article denounces.

if you wanted to just fetch data from your server, it's not a problem anyway, right? a spa can also just fetch fresh data from a server. the whole point of the frontend cache was optimising ux/latency, e.g. for apps with global users but not globally deployed servers

everything else is simply HTML,

and yes, the article is targeting the SPA crowd

1. always refetch data. always in sync, but needs a server request, so it's slow.

2. cache some data on the client. faster, but you're "building your own database". can get out of sync (redux)

3. NEW: use SQLSync. fast, client & server stay in sync, don't have to "build your own database"

what you're describing just seems like number 1, right?

Except you're not fetching data in htmx, you're fetching hypertext. This is an important distinction. With a JS app you fetch data, integrate it into some local data structure that is then rendered into some form that generates HTML that the browser then renders. With htmx, all of those steps happen in the (faster) server, and the client only receives and renders the HTML. The client-side steps are faster.

Furthermore, apps end up structured differently because you're not querying fine-grained data.

[0] https://typeorm.io/ [1] https://typeorm.io/supported-platforms#browser

SQL.js is an inspiring project in the space and led to official upstream support for compiling SQLite to Wasm. Without these projects SQLSync would have been much more difficult (akin to the original difficulty of SQL.js in the first place). That said, SQLSync is also unique from SQL.js in that it includes a synchronization layer that coordinates with the SQLSync server to provide real time collaboration between users.

The reason we aren't using Sqlite for our product is because Sql is frankly not the right tool for querying data for an application. It doesn't easily map to the data-structures you want in your client code and nearly all SQL databases have no way to subscribe to changes to a query without polling the query repeatedly.

So if you like the idea of having a complete database on your client but also want deep integration with Typescript/Javascript check out what we're building at https://github.com/aspen-cloud/triplit

https://github.com/cpursley/walex

I disagree. Normalizing data is critical for FE reactive applications, keeping data up-to-date basically requires it; all CRUD operations are much easier to handle.

But how does it compare to ElectricSQL[1] and PowerSync[2]?

[1] https://electric-sql.com/ [2] https://powersync.com/

ElectricSQL and PowerSync are both tackling the very hard problem of partial replication. The idea is to build a general solution which allows a traditional centralized db to bidirectionally sync only what's needed on the client side - while still supporting optimistic mutations (and all the consistency/conflict stuff that goes along with that).

The downside is implementation complexity. Both require the ability to keep track of precisely the set of data on each client in order to push out changes to only that subset of the overall database. In addition, specifying which subsets of the database state to pull down requires a new DSL and is a new thing to learn (and optimize). That said, I'm stoked they are taking on this extremely hard problem so when SQLSync is ready for partial replication someone will have already figured out the best practices.

SQLSync, on the other hand, only supports full db sync. So every client will see a consistent view of the entire database. You might immediately wonder if this is a good idea - and for some apps, it's not. But consider a personal finance app. The main goal is cross device sync, cloud backup, offline capable, etc. In this case having the entire db stored on every device is probably what you want. Another example is a document oriented data model, such as Airtable. Each Airtable could be a distinct database, thus leaving it up to the client to manage which tables they care about.

(added in edit:) By focusing on full db sync, the sync engine is much simpler than solutions that support partial replication. One benefit of this is that the backend is very lightweight. Currently the demo (https://sqlsync-todo.pages.dev) runs entirely within Cloudflare Durable Objects using very little storage and CPU time.

SQLSync has a ton of work to do to make these use cases possible (still very much a prototype), but my initial tests have been extremely promising. Hope this helps!

(edit: clarified language regarding centralized dbs and full db sync. Also added paragraph regarding full db sync)

A few things to clarify:

>one multi-tenant centralized db to bidirectionally sync

PowerSync supports syncing from multiple databases.

>The downside is complexity.

I'd say this is true if you're building a partial replication system yourself. PowerSync gives you a ready-built system that's been proven at scale and therefore lets you avoid most of that complexity.

>SQLSync, on the other hand, is full db sync.

It's just as easy to sync the full db with PowerSync as do partial sync.

Edit: formatting

I've adjusted my comment to be more clear and hopefully more fair. I didn't mean to mis-imply anything about your service.

A bit confused by this. If I'm a developer of a PFM, I don't want anything but a single user's financial data synced to their device. This sounds like partial replication to me.

(Note: currently SQLSync's server tier doesn't support auth, just random 128bit ids. Auth will come as it matures - but don't use this for anything super secure at the moment).

> Step 2: Install and configure the React library

I really need to write up a detailed overview of how this works! Thanks for the feedback!

I do need better in the moment state in the client though. Was looking at react query with perhaps websockets for cache invalidation. It's nice to see this sqlsync idea too though to consider.

Is the typical in-browser code really complex enough that one would want to introduce the object-relational mismatch and all of its associated issues?

Most in-browser code wants to operate on a limited set of very well-defined tree data structures, which is a bad fit for relational databases because relational databases are designed to represent a generic graph that can be virtualized into a tree arbitrarily and as necessary.

https://news.ycombinator.com/from?site=sqlsync.dev

Admittedly, I haven't yet read the linked article. But, I plan to.

Perhaps in the future, SQLSync will support partial replication which would enable this use case. As always there are trade-offs to consider with both approaches.

There are hundreds of subtly different database out there, but there are only a dozen or two that really ought to exist. So stop making new databases for every little thing folks. They are

I get that you can sync state between browser tabs, but I'm trying on two different devices (iPhone and Desktop).

And as far as I can tell, the Wasm layer can't perform network requests directly.

UPDATE: In the console tab I can see 'coordinatorUrl: 'wss://sqlsync.orbitinghail.workers.dev', but I was expecting to see this Websockets connection in the Network tab, and it isn't.

I built a support chat app with it (which we use ourselves) using only declarative HTML components. Supports both public chat and private chat with authentication.

It's fully working now but I'm now focusing on walkthroughs/guides.

https://saasufy.com/

I suspect you would be better off by creating a rust reducer trait, and lifting that wasm-in-wasm complexity into a new crate implementing the reducer trait through wasm-in-wasm for the people who want that.

But maybe i'm missing something.

As for wasm-in-wasm specifically. It would be nice to experiment with the component model to load reducers alongside SQLSync - but the UX isn't quite there yet.

edit: I mean rebase, not CRDT

Basically each client maintains their own log of mutations. These mutations are applied optimistically to the local instance of SQLite, and in parallel we replicate the log to the server.

On the server side, it reads from all of the individual client logs in a deterministic order and applies the mutations to it's own SQLite database. Under the hood SQLSync hijacks all of the writes to storage and organises them into a format that's easy to replicate.

Finally, the the storage log from the server is replicated back down to each of the clients. This leaves the clients in a weird position as they have two versions of the database which may have diverged (probably). So to finish this up, the clients throw away their local changes and reset to the server state (i.e. git reset --hard). Then since there may be some mutations that were executed on the client but not yet run on the server the client simply re-runs those mutations (i.e. git rebase).

In this way the system continuously keeps itself in sync with the server and other clients.

Conflicts are handled by logic in the reducer which is able to inspect the state of the database to figure out what to do. Yes, this does require writing the reducer code carefully - however in testing I've found this not to be too bad because:

1. A lot of SQL operations automatically converge pretty nicely (and you probably already have to think about them converging in your existing REST api's or w/e backend api you are writing). Think about patterns like `insert...on conflict do...` for example.

2. Since the reducer is isolated and comes with a type description of all possible mutations, it's very easy to unit test the reducer with different orderings. Basically think of it as deterministic simulation testing for your API. Something that's pretty hard to do with normal backend architectures without a lot of mocking and architecture.

Hopefully that helps!

https://guides.emberjs.com/release/models/

There’s also https://orbitjs.com/

The article is responding to the pattern of yet another custom data model and custom data API (à la Ember).

Instead provide an SQL database (the well proven SQLite) within the front end and use SQL to interact. And sync data from-to the backend DB.

Which one could then slap on a model or ORM layer on top of - should that be one's bent.

It isn't clear how they manage the subscription to data updates/inserts/deletions - it mentions supporting triggers, but that feels icky to me.

> It isn't clear how they manage the subscription to data updates/inserts/deletions - it mentions supporting triggers, but that feels icky to me.

Architecture post coming soon. In the meantime, I want to clarify that SQLSync does not use triggers for sync. Instead, I hijack SQLites page storage and added page replication to it. Writes are consolidated through an API I call the "reducer" which allows SQLSync to keep track of which logical writes correspond to which sets of page changes. The actual sync is pretty dumb: we run the reducer on both the client and the server. The client replicates down server pages, and then periodically throws out local changes, resets to the server state, and then replays any mutations that haven't yet been acked on the server.

And that's despite me having worked with Cassandra, Kafka, Postgres and a variety of programming languages, DevOps tools, having worked with Vuejs and React.

- page storage

- reducers

- replaying mutations

- acks

you're unclear on?

It's called a reducer because it operates like a reduce/fold operation, in that if you take a given state and an ordered list of mutations, you'll end up with a new state out the other side, much like 'let newState = list.reduce(reducer, initialState)' would in JS.

The reducer model is how Elm's state management works, and Redux's, and is what allow replaying of subsequent changes against a previous state to get to a current state (which enables, amongst other things, some the features of those systems' respective dev tools).

The article links to the (nearly trivial) reducer for the todo example, which is on github here: https://github.com/orbitinghail/sqlsync/blob/ba762ce2a10afbb...

EmberData, Ember Routing, Ember Multi Applications (can't remember what its called, but its a precursor to microfrontends) all in one CLI tooling etc.

I could never understand what holds Ember back from being more used. I think it used to be performance but I think they addressed that with Glimmer many years ago.

It's not being used more for the same reason Ruby on Rails is not used more.

Both are batteries included frameworks, with a much steeper learning curve than e.g.

Plus people like to tinker and wire up things together themselves, like router, rendering, state management, testing, etc. That's more exciting than `ember new project`.

I was like that 10 years ago too, but now when I learned the ropes I just want to focus on shipping and providing value, rather than wasting my life wiring up 17 javascript packages of the month together.

  There comes a time [..] where we [..] need to cache data from an API. It might start off benign – storing a previous page of data for that instant back button experience, implementing a bit of undo logic, or merging some state from different API requests.

  SQLSync is [..] designed to synchronize web application state between users, devices, and the edge.

Can we not just admit the browser is a shitty client app? I get that like 50% of the software developers in the world have made the browser their literal livelihood and career, but jesus christ, we need to end this madness. Either make a full blown VM and just write normal client applications that run in it, or stop trying to make client applications out of javascript.

It's insane how much of the world's engineering effort and technology investment money has gone into putting lipstick on a pig. If we took all the money invested in browser ecosystems and put it towards spaceflight, we'd be living on Mars right now.

And now we see the beginning of a migration back to client-side computation and storage.

Somehow, though, this latest iteration doesn't make a lot of sense. It's hard enough maintaining data consistency on a web service that may be used by hundreds or thousands of people. Imagine when this data is cached in microdatabases in unreliable browsers.

On top of that, the browser makes an absolutely horrible programming environment for client-side apps. For the programming part, Javascript is an poor language, so you wind up using heavy-duty frameworks like React to make it tolerable. For the UI representation, that's just not what HTML/CSS were ever meant for. So you get frameworks there as well. Young developers think this is just the way it is. No actually, it's more like the worst of all possible worlds. Using something like JavaFX or (I know, I know) even Visual Basic, you can produce a functional, robust UI with a tiny fraction of the effort.

They made an observation that our industry goes in cyclical centralize/de-centralize cycles and that we we were (at the time) entering into a centralization cycle.

Now here I am reading a comment that we're going back into a de-centralization cycle and I wouldn't be surprised if you're the same poster.

probably 15-20 years ago (maybe more?) I made a prediction that I still think will come true.

The OS will become the "browser" and applications will run directly on the OS and will access local resources through standardized interfaces. WebAssembly and things like WebGL are already moving us in that direction. Honestly HTML5 was the first time I recognized standard updates as moving us towards that reality with things like localStorage, etc.

I honestly think if someone more imaginative had the helm at MS when the cloud started getting big they would have eaten google's lunch by leveraging their desktop dominance into the web. Instead they dd inane things like display websites on the desktop (win98 IIRC).

You may be right about the browser becoming the OS. Chromebooks were already a step in that direction. But JS/HTML/CSS really is a horrible combination for application programming. If the browser does become the OS, can we please get decent technology to work with?

They are bad at just being directly programmable targets.

This is actually happening, albeit slowly, with recent efforts around WASM etc. If you want a fun hypothetical of where this all goes, check out the talk "The Birth & Death of JavaScript". Link here: https://www.destroyallsoftware.com/talks/the-birth-and-death...

But consider how WASM is now turning the browser into an app delivery client. Not a “html and json bodged into an app”, but a real honest to god app.

This project happens to be browser based because that’s convenient place to put a WASM app, and it has a decent presentation layer. But it doesn’t have to be!

They didn't originally: Java and ActiveX originally weren't sandboxed and had free run of the visitor's computer.

All major OSes today now have built-in support for process/app sandboxing. I suppose if the "rich client" frontend model (WPF, etc) was more popular then I expect desktop OS application isolation to have been introduced much sooner.

Security development happens where the market demands it, and rarely does it happen where it's actually needed.